7 ms·
Semantic search engine for ArXiv, biorxiv and medrxiv
- elliotec 1y agoThis is really cool, and very relevant to something I'm working on. Would you be willing to do a quick explanation of the build?
- 0101111101 1y agoSure! I first used openai embeddings on all the paper titles, abstracts and authors. When a user submits a search query, I embed the query, find the closest matching papers and return those results. Nothing too fancy involved! I'm also maintaining a dataset of all the embeddings on kaggle if you want to use them yourself: https://www.kaggle.com/datasets/tomtum/openai-arxiv-embeddings/data https://www.kaggle.com/datasets/tomtum/openai-arxiv-embeddin...
- heisenburgzero 1y agoSo did you just combine Title+Abstracts+Authors into a single chunk and embed them or embedded them individually?
- synctext 1y agoImpressive! Will you parse the papers in the future? Without citations this is not that usable for professors or scientists in general. The relevance ranking largely depends on showing these older, prominent papers. (from our lab experience building decentralised search using transformers)
- 0101111101 1y agoOne chunk embedded together
- elliotec 1y agoThank you!!
- cluckindan 1y agoThat method can break when author names and subject matter collide.
- 0101111101 1y agoTrue, but similarly if your embeddings are any good they'll capture interesting associations between authors, topics and your search query. If you find any interesting author overlap results I'd be very interested!
- cluckindan 1y agoNot exactly what I was looking for, but interesting nonetheless: https://arxivxplorer.com/?q=exotic+penis https://arxivxplorer.com/?q=exotic+penis
- fdadsfdsd 1y ago[flagged]
- madars 1y agoLooks great! Could you add eprint.iacr.org (Cryptology ePrint Archive)?
- 0101111101 1y agoDo they have a public API/dataset?
- madars 1y agoThey have RSS feeds for new/updated papers: https://eprint.iacr.org/rss/ https://eprint.iacr.org/rss/
- faffdsf 1y ago[dead]
- bbor 1y agoOh god, there's a medrxiv?? TIL... Don't forget chemrXiv!
- 0101111101 1y agoSadly I couldn't find a public API for chemrxiv, but would be happy to be proven wrong!
- BioGeek 1y agoHere it is: https://chemrxiv.org/engage/chemrxiv/public-api/documentation https://chemrxiv.org/engage/chemrxiv/public-api/documentatio...
- 0101111101 1y agoThanks!
- dhacks 1y agoThere is also engrXiv, which has an OAI endpoint. https://engrxiv.org/oai?verb=ListRecords&metadataPrefix=oai_dc https://engrxiv.org/oai?verb=ListRecords&metadataPrefix=oai_...
- 0101111101 1y agoAmazing!
- Fomite 1y agomedrxiv was very useful for keeping the various COVID-19 related preprints from completely swamping biorxiv, especially once biorxiv started aggressively rejecting them.
- sitkack 1y agohttps://en.wikipedia.org/wiki/List_of_preprint_repositories https://en.wikipedia.org/wiki/List_of_preprint_repositories
- sitkack 1y agoembedding search via https://searchthearxiv.com/ https://searchthearxiv.com/ takes either a word vector, or an abs or pdf link to an arxiv paper. https://news.ycombinator.com/item?id=42519487 https://news.ycombinator.com/item?id=42519487 I just did a spot check, I think searchthearxiv search results are superior.
- 0101111101 1y agoLooks cool! You can input either a search query or a paper URL on arxiv xplorer. You can even combine paper URLs to search for combinations of ideas by putting + or - before the URL, like `+ 2501.12948 + 1712.01815`
- sitkack 1y agoThat is neat I like that. It would be cool if the "More Like This" had a + button that would append the arxiv id to the search query.
- 0101111101 1y agoThat's a nice idea! Might take a look this weekend!
- masterjack 1y agoThere’s also the search and browsing on https://sugaku.net https://sugaku.net, it’s more focused on math but does also have all of the arxiv on it
- nblgbg 1y agoJust curious, are there any techniques other than using embeddings, computing cosine similarity, and sorting the results based on that? RRF could be used but again its very simple as well.
- forrestp 1y agoMy understanding is that your levers are roughly better / more diverse embeddings or computing more embeddings (embed chunks / groups / etc) + aggregating more cosine similarities / scores. More flops = better search w/ steep diminishing returns Colbert being a good google-able application of utilizing more embeddings. Search ends up often being a funnel of techniques. Cheap and high recall for phase 1 and ratchet up the flops and precision in subsequent passes on the previous result set.
- 0101111101 1y agoExactly! A near property of the matryoshka embeddings is that you can compute a low dimension embedding similarity really fast and then refine afterwards.
- itzlambda 1y agoUsing "+" and "-" in search is interesting idea. I've built similar thing for github stars[1], might implement the same for it. [1]: https://starscout.xyz/ https://starscout.xyz/