5 ms·
Show HN: Deep search of all ML papers
Built an automated system to run a deep search of ArXiv and carefully find all the precise papers that exist on a complex topic.
It's different from simple RAG because it searches, classifies, and adapts based on relevant papers it uncovers, and then continues until it finds every paper on a topic (trying to mimic the human research process). Benchmarked 10x higher accuracy and total retrieval compared to Google Scholar for a median search (whitepaper on website). Also knows when it is complete, and misses virtually nothing (< 3% or so, once it's converged).
Website has a free trial and a bunch of example search reports. Want feedback and suggestions.
- tomhartke 3y agoHere's an example report on: tokenization-free large language model architectures, which have been shown to achieve compute/accuracy tradeoffs comparable to or better than traditional token-based models https://app.undermind.ai/query_app/display_one_search/05f0b8be99236b3a600a85f67de19284a0b34aecf5eb593c163d9be1e6249652/ https://app.undermind.ai/query_app/display_one_search/05f0b8...
- axpy906 3y agoWhat about compare to ArXiv Sanity Preserver?
- axg11 3y agoI would change the main CTA to "Try it now" and then use a different style for "Read the stats". It currently looks like there are two equally important CTAs. If you can find a way to make the results closer to real-time, this will be a really popular product.
- tomhartke 3y agoAppreciate the advice. Re: timing, it's bottlenecked by the sequential nature of the search. To be comprehensive, we discover a few papers, and use that info to choose where to look more closely next.
- knicholes 3y agoImmediately bailed once it required I provide my email address.
- deleted 3y ago[deleted]
- huqedato 3y agoNice but way too pricey. https://chat.openai.com/g/g-dGz4aw9iA-research-refiner https://chat.openai.com/g/g-dGz4aw9iA-research-refiner - the free version (just ChatGPT Plus subscription needed)
- canadiantim 3y agoAwesome had no idea this existed. Very useful, thanks!
- notso411 3y ago[dead]
- tomhartke 3y agoThe goal is to be systematic and handle complex topics. ChatGPT + keyword search can't handle complex topics at all, and isn't systematic either.
- danielmarkbruce 3y agoIt's still early days. Many folks think that GPT-4 + simple RAG works well. It doesn't. Building a good tool like this is hard.
- 3abiton 3y agoI wonder how many of such services chatgpt undercuts already?
- callalex 3y agoFree, just needs a subscription?
- wackget 3y agoThere's something disagreeable about charging a subscription to search freely-available scientific papers. Yeah I get you're technically paying for the "advanced" search but it still leaves a bad taste in the mouth because this service's entire existence depends on open source knowledge. P.S. hiding pricing behind registration isn't cool
- deleted 3y ago[deleted]
- danielmarkbruce 3y ago[flagged]
- bbsz 3y agoI think that full text search queries over long text data is already kind of expensive server side. Users are paying specifically for this, not better UI or simple direct match search available in free to use projects. I would say it's very reasonable to charge for costs incurred here.
- kingkongjaffa 3y agoIt's a shame the research publishing industry is a bunch of walled gardens. Since this only supports arXiv, and not paper repositories from other industries.
- jakderrida 3y agoMaybe the model will get smart enough to go to SciHub and Libgen? IP holders and distributors come after me with evidence, I'll just pull out my belt and tell them I gotta go teach some naughty GPUs another lesson.
- whalekawhi 3y ago[dead]
- krohling 3y ago"Our AI agent finds precisely what you ask for, 10-50x better than Google Scholar" I was curious how this was measured since benchmarking accuracy for LLMs is tough. Found this in the paper: "This classification accuracy was benchmarked by manually analyzing over 400 papers across a range of representative searches, and comparing the human evaluation to the language model’s judgment" I'm skeptical that their dataset of 400 papers with 3 classification labels (highly relevant, closely related, or ignorable) is large enough to represent the diversity of queries they're going to get from users. To be clear, I don't think this undermine's (haha) the value of what they've built, still very cool.
- basb77 3y agoHow does this compare to a system like Elicit? Seems to be very similar at first glance.
- tomhartke 3y agoFYI system is a bit delayed because of traffic levels. May take a bit longer to generate results at the moment (usually takes ~10 min).
- frogamel 3y agoIMO if you're going to profit off open research, you should at least make your own work available for other researchers. The white paper has 10 pages of performance benchmarks but 5 sentences on methodology.
- deleted 3y ago[deleted]
- espadrine 3y agoIt is certainly interesting, and I would love to try it for my hobbyist use-cases. I don’t do much research at work, but a fair bit on the weekends. Are you filtering users however? I cannot sign up in a personal capacity with a GMail email. The page raises this error: “Please use a valid institutional or company email address.”
- htrp 3y agowhat is pricing?
- WhitneyLand 3y agoI don’t want to be negative but you asked for feedback, so I’ll give you a few impressions including the superficial and quite subjective fwiw: 1. The hyperbolic claims are going to be off-putting to some. You’ve “solved” ML search? 50x better than Google scholar on a metric no one’s been benchmarking against? Consider your audience and what they would find credible. 2. The UX needs work. To give one aesthetic example, in the results there are large, brightly colored, red and green circles that are used inconsistently, and they clash the palette. This stuff can affect how sticky your service is. 3. Don’t restrict signup by email domain. This is nuts. Never add friction to gaining customer relationships. If you’re capacity constrained limit the trial. If you’re trying to segment the market there are better ways. 4. The name “Undermind”, is not working to my ear. It’s worth changing. At least find a product person whose opinion you respect and ask their take. 5. I think a lot of people here would be willing to give you useful technical feedback on the architecture and approach if more information were shared about how the service works, but I didn’t notice that was available.
- mangoo84 3y ago[dead]
- Yenrabbit 3y agoI tested this out on a topic I'd been discussing with some fellow researchers, and it pulled in the papers we'd chatted about plus a bunch of related ones that look very relevant and interesting. Congrats on a cool project!