7 ms·

Show HN: I mapped 8.5M research papers into an interactive atlas

85 points by leonickson 2mo ago

When I read papers, I have to jump between multiple tabs to find the dataset, code, videos, peer reviews, and so on. I tried to fix this with this project. It started as a project just for papers on arXiv, but after its initial success on Twitter (got like 1.9k views: the most I have gotten for a post), I have now expanded it to include other openly available papers from PubMed Central, bioRxiv, medRxiv, and eLife. These papers have been linked with their genes, proteins, diseases, drugs, clinical trials, 3D protein structures, code, and cited and similar papers. This project now has four parts: First, a map. I embedded nearly 8.5M papers (with SPECTER2), ran UMAP for 2D representation, and rendered them as a scatterplot. The dots can be clicked to see brief information about the papers, like an LLM TLDR, key findings, peer reviews, linked entities, and more. The clusters are also labeled, though you might have to zoom in. Second, I built a detailed paper page for each paper. They give you the paper's full text, images, videos, peer reviews (from OpenReview), GitHub links, Hugging Face dataset/model links, clinical trials, genes, diseases, 3D protein structures, cited papers, and similar papers. You can also copy the whole page, including the full paper text and image URLs, as markdown for your LLM. Third, I have released an extension so you can read all this information in your sidebar by clicking "open in Tomesphere" that shows up in arXiv, PMC, bioRxiv, Google Scholar, or medRxiv. I have tried to provide as much information as possible in the extension, though for things like viewing all the images or a 3D protein structure, you might still have to go to the paper page using the link provided in the extension. Fourth, all this data is available for your LLM via MCP. The MCP does have a 50-query free limit (this jumps 10x with signup). Note: this project is still in beta, so papers might have some mismatched information. I am rolling out feedback forms soon to improve the data quality. Thank you so much for taking the time to read this.

deleted 2mo ago
[deleted]
murkt 2mo ago
I remember similar kind of visualization from a decade ago, called paperscape. Looked cool, worked on clustering using citations and references. Never got any idea on any use case that would be covered by such visualizations, apart from looking cool.
- addycb 2mo ago
  That's usually the case with graph visualizations or clustering for networks, imo (beyond revealing obvious statistics(
  - specproc 2mo ago
    I love them! It's a really nice, fun way to explore a corpus. Cosmograph for this sort of thing is great, it supports graphs as well as 2D projections, and is blazing fast. That said, I've never had a client or stakeholder show any interest in using one, beyond an initial "that's cool". And UMAP etc., is just as much an art as a science. You'll go mad trying to get the perfect layout. Great toy if you're into that sort of thing, but yeah, fiddly and overwhelming for most.
    - leonickson 2mo ago
      Hi, I love the genre too. Cosmograph is wonderful, I did try it, but because of its license restriction I could not use it for this project. I do agree that beyond an initial "that's cool" this map may not contribute much, "and that's why I didn't make it the main product. I already had the data as I was building other things (extension, paper page) and wanted a bit of a cool factor so people would take a look at the project. The value is what's under each dot, the enriched page (TLDR, genes/drugs/diseases, trials, 3D structures, code, datasets, full text), extension and the MCP for agents.
- gavinray 2mo ago
  ResearchRabbit is free and has this feature! https://www.researchrabbit.ai/ https://www.researchrabbit.ai/ ConnectedPapers also has this but they started to limit unless you pay: https://www.connectedpapers.com/ https://www.connectedpapers.com/ A few other ones I know of: https://litmaps.com https://litmaps.com https://consensus.app/home/features/citation-graph/ https://consensus.app/home/features/citation-graph/
  - aziis98 2mo ago
    I also made my own variant recently. I really liked the idea of litmaps but I didn't really like the UI/UX or "graph expand" feature (it uses some internal heuristic that is not very clear). https://paper-explorer.aziis98.com/ https://paper-explorer.aziis98.com/ (also on github https://github.com/aziis98/paper-explorer https://github.com/aziis98/paper-explorer) This uses OpenAlex as a source of articles and to let you explore the citation graph of papers. This is still a prototype mostly made to test out how far I could go with vibecoding (well I still checked the code now and then) something without a js framework. Someday I think I will add more features to it, but now there is already a somewhat working version of import/export so I'm fine with it.
gavinray 2mo ago
Neat! Two questions I had after using it: 1) Is there a way to filter the visual atlas by the search term? For instance, I searched "ribosome" and it gave me a list, but I couldn't seem to visualize the list 2) I notice there's an MCP tool. I've used https://paperclip.gxl.ai/ https://paperclip.gxl.ai/ in the past to good effect, curious if there are any standout features from tomesphere?
- leonickson 2mo ago
  The project has four parts, and I think you may have used the search in the navigation bar. That search is for the paper page (detailed information about that specific paper). When you search, you get a list of results, and if you click one of them, it takes you to a paper page with all the in-depth details about that paper. I should have made the map UX better. To highlight things in the atlas or map, you might want to go to the filter panel on the left side and scroll down a bit. You will see a search area that helps you search for genes, diseases, and proteins. However it might not highlight any dot for the “ribosome” because filter search for now is only connected to genes, diseases and protein. I noted this, and I will improve it. I may also move that filter search to a different place. Thank you so much. Tomesphere includes web pages and a browser extension overlays all of this directly on the arXiv, PMC, bioRxiv, Google Scholar, and medRxiv pages you are already reading. I had noticed the Paperclip MCP before, and from what I can see, they have very good data. In some cases, they may even have better data. We also have some additional sources, such as peer review from OpenReview, video links from YouTube and SlidesLive, GitHub links, AlphaFold protein entities, citations, and semantic neighbors. Thank you for the questions. Will improve the project more.
vignyBot 2mo ago
Just curious where do you source all those papers?
- leonickson 2mo ago
  It is all open, bulk-downloadable sources. Here are some links. - arXiv: bulk on S3 (https://info.arxiv.org/help/bulk_data_s3.html) - PubMed Central Open Access: AWS Open Data (s3://pmc-oa-opendata). - bioRxiv / medRxiv: their monthly S3 dumps - OpenAlex: for citations, metadata, abstracts (https://developers.openalex.org/download/download-to-machine) - AlphaFold — structures (https://alphafold.ebi.ac.uk/download) Thanks for the question.
goodwillhunting 2mo ago
this is fun! even for casual users. I searched "James Webb" and got 'James Webb Space Telescope: data, problems, and resolution A. D. Dolgov · 2023 · 1 cites' as my first result, but clicking it didn't do anything - I expected it to either zoom into or see the article contents. Was I doing something wrong? (Chrome, OSX).
- leonickson 2mo ago
  I am sorry, it should have taken you to the paper page/article, I will investigate it, but I did the same search and clicked on the first link, it to me to this - https://tomesphere.com/paper/2301.01365 https://tomesphere.com/paper/2301.01365 Thanks for letting me know.
VinayUPrabhu 2mo ago
This is amazing! Would love to collaborate on a position paper I've been authoring if you have bandwidth
- leonickson 2mo ago
  Thank you so much! I'd genuinely like to hear more, you can email me here - help@tomesphere.com
GymInMeCricket 2mo ago
I have a few questions: 1. I see there is a Chrome extension, but I have not used Chrome since their adblocker blocker announcement. Is a Firefox extension planned? 2. What is the business model? Is this an open beta of a paid product? If this is not a product, will the code be released at some point? 3. It would be helpful to be able to filter papers by institution or author. Is this planned or out of scope?
- leonickson 2mo ago
  Hello, 1. Yes, I am working on it, I will soon release it for Firefox and Safari (by this weekend). 2. For now I am just trying to see what people want, may be the business model is more focused on mcp for higher limits for labs, lets see where is goes. 3. The paper/article search already has author search, I can add the university search soon in both the paper/article and altas/map search. Thank you for all the questions and suggestions.
- leonickson 2mo ago
  The extension is live in Firefox - https://addons.mozilla.org/addon/tomesphere/ https://addons.mozilla.org/addon/tomesphere/
ColinEberhardt 2mo ago
Very cool. By the way, you can render many more datapoints on mobile if you use WebGL. Here’s a similar example - embeddings rendered using a T-SNE layout https://blog.scottlogic.com/2021/10/15/efficiently-loading-massive-d3-datasets-using-apache-arrow.html https://blog.scottlogic.com/2021/10/15/efficiently-loading-m...
- leonickson 2mo ago
  Will check it, it will be really useful. Thanks
mfscholar 2mo ago
For your reference, check the 3M map of science papers we made here: https://www.scholar-inbox.com/scholar-maps https://www.scholar-inbox.com/scholar-maps
- leonickson 2mo ago
  I really like your work with the gradients.
hadi121 2mo ago
This is super cool. Just one question. I noticed that papers have different spacing between them. Is that because a paper's placement is based on correlation with neighbors, or just random placement?
- leonickson 2mo ago
  [dead]
abhiram304 2mo ago
[flagged]
Kerbonut 2mo ago
Tailscale Aperture is having trouble doing the OAUth2 connection. Can you support oauth2_dcr or test with Aperture?
- leonickson 2mo ago
  Yes, DCR is supported (WorkOS exposes the registration endpoint). What error is Aperture throwing? I suspect a metadata-discovery detail.
rahulb0802 2mo ago
I really like the clustering aspect of this. It would be cool if there was a way to articulate some sort of relationship between two adjacent or nearby papers in this tool.
Jahnavi_Aleti 2mo ago
Jumping between arXiv, GitHub, and peer reviews is such a real pain. The “copy page as markdown for your LLM” feature is especially useful.Curious how accurate the entity linking is so far, genes/diseases/etc. That’s usually where these tools get messy.

85 points by leonickson 2mo ago

deleted 2mo ago

[deleted]

murkt 2mo ago

I remember similar kind of visualization from a decade ago, called paperscape. Looked cool, worked on clustering using citations and references. Never got any idea on any use case that would be covered by such visualizations, apart from looking cool.

addycb 2mo ago
That's usually the case with graph visualizations or clustering for networks, imo (beyond revealing obvious statistics(
- specproc 2mo ago
  I love them! It's a really nice, fun way to explore a corpus. Cosmograph for this sort of thing is great, it supports graphs as well as 2D projections, and is blazing fast. That said, I've never had a client or stakeholder show any interest in using one, beyond an initial "that's cool". And UMAP etc., is just as much an art as a science. You'll go mad trying to get the perfect layout. Great toy if you're into that sort of thing, but yeah, fiddly and overwhelming for most.
  - leonickson 2mo ago
    Hi, I love the genre too. Cosmograph is wonderful, I did try it, but because of its license restriction I could not use it for this project. I do agree that beyond an initial "that's cool" this map may not contribute much, "and that's why I didn't make it the main product. I already had the data as I was building other things (extension, paper page) and wanted a bit of a cool factor so people would take a look at the project. The value is what's under each dot, the enriched page (TLDR, genes/drugs/diseases, trials, 3D structures, code, datasets, full text), extension and the MCP for agents.
gavinray 2mo ago
ResearchRabbit is free and has this feature! https://www.researchrabbit.ai/ https://www.researchrabbit.ai/ ConnectedPapers also has this but they started to limit unless you pay: https://www.connectedpapers.com/ https://www.connectedpapers.com/ A few other ones I know of: https://litmaps.com https://litmaps.com https://consensus.app/home/features/citation-graph/ https://consensus.app/home/features/citation-graph/
- aziis98 2mo ago
  I also made my own variant recently. I really liked the idea of litmaps but I didn't really like the UI/UX or "graph expand" feature (it uses some internal heuristic that is not very clear). https://paper-explorer.aziis98.com/ https://paper-explorer.aziis98.com/ (also on github https://github.com/aziis98/paper-explorer https://github.com/aziis98/paper-explorer) This uses OpenAlex as a source of articles and to let you explore the citation graph of papers. This is still a prototype mostly made to test out how far I could go with vibecoding (well I still checked the code now and then) something without a js framework. Someday I think I will add more features to it, but now there is already a somewhat working version of import/export so I'm fine with it.

gavinray 2mo ago

Neat! Two questions I had after using it: 1) Is there a way to filter the visual atlas by the search term? For instance, I searched "ribosome" and it gave me a list, but I couldn't seem to visualize the list 2) I notice there's an MCP tool. I've used https://paperclip.gxl.ai/ https://paperclip.gxl.ai/ in the past to good effect, curious if there are any standout features from tomesphere?

leonickson 2mo ago
The project has four parts, and I think you may have used the search in the navigation bar. That search is for the paper page (detailed information about that specific paper). When you search, you get a list of results, and if you click one of them, it takes you to a paper page with all the in-depth details about that paper. I should have made the map UX better. To highlight things in the atlas or map, you might want to go to the filter panel on the left side and scroll down a bit. You will see a search area that helps you search for genes, diseases, and proteins. However it might not highlight any dot for the “ribosome” because filter search for now is only connected to genes, diseases and protein. I noted this, and I will improve it. I may also move that filter search to a different place. Thank you so much. Tomesphere includes web pages and a browser extension overlays all of this directly on the arXiv, PMC, bioRxiv, Google Scholar, and medRxiv pages you are already reading. I had noticed the Paperclip MCP before, and from what I can see, they have very good data. In some cases, they may even have better data. We also have some additional sources, such as peer review from OpenReview, video links from YouTube and SlidesLive, GitHub links, AlphaFold protein entities, citations, and semantic neighbors. Thank you for the questions. Will improve the project more.

vignyBot 2mo ago

Just curious where do you source all those papers?

leonickson 2mo ago
It is all open, bulk-downloadable sources. Here are some links. - arXiv: bulk on S3 (https://info.arxiv.org/help/bulk_data_s3.html) - PubMed Central Open Access: AWS Open Data (s3://pmc-oa-opendata). - bioRxiv / medRxiv: their monthly S3 dumps - OpenAlex: for citations, metadata, abstracts (https://developers.openalex.org/download/download-to-machine) - AlphaFold — structures (https://alphafold.ebi.ac.uk/download) Thanks for the question.

goodwillhunting 2mo ago

this is fun! even for casual users. I searched "James Webb" and got 'James Webb Space Telescope: data, problems, and resolution A. D. Dolgov · 2023 · 1 cites' as my first result, but clicking it didn't do anything - I expected it to either zoom into or see the article contents. Was I doing something wrong? (Chrome, OSX).

leonickson 2mo ago
I am sorry, it should have taken you to the paper page/article, I will investigate it, but I did the same search and clicked on the first link, it to me to this - https://tomesphere.com/paper/2301.01365 https://tomesphere.com/paper/2301.01365 Thanks for letting me know.

VinayUPrabhu 2mo ago

This is amazing! Would love to collaborate on a position paper I've been authoring if you have bandwidth

leonickson 2mo ago
Thank you so much! I'd genuinely like to hear more, you can email me here - help@tomesphere.com

GymInMeCricket 2mo ago

I have a few questions: 1. I see there is a Chrome extension, but I have not used Chrome since their adblocker blocker announcement. Is a Firefox extension planned? 2. What is the business model? Is this an open beta of a paid product? If this is not a product, will the code be released at some point? 3. It would be helpful to be able to filter papers by institution or author. Is this planned or out of scope?

leonickson 2mo ago
Hello, 1. Yes, I am working on it, I will soon release it for Firefox and Safari (by this weekend). 2. For now I am just trying to see what people want, may be the business model is more focused on mcp for higher limits for labs, lets see where is goes. 3. The paper/article search already has author search, I can add the university search soon in both the paper/article and altas/map search. Thank you for all the questions and suggestions.
leonickson 2mo ago
The extension is live in Firefox - https://addons.mozilla.org/addon/tomesphere/ https://addons.mozilla.org/addon/tomesphere/

ColinEberhardt 2mo ago

Very cool. By the way, you can render many more datapoints on mobile if you use WebGL. Here’s a similar example - embeddings rendered using a T-SNE layout https://blog.scottlogic.com/2021/10/15/efficiently-loading-massive-d3-datasets-using-apache-arrow.html https://blog.scottlogic.com/2021/10/15/efficiently-loading-m...

leonickson 2mo ago
Will check it, it will be really useful. Thanks

mfscholar 2mo ago

For your reference, check the 3M map of science papers we made here: https://www.scholar-inbox.com/scholar-maps https://www.scholar-inbox.com/scholar-maps

leonickson 2mo ago
I really like your work with the gradients.

hadi121 2mo ago

This is super cool. Just one question. I noticed that papers have different spacing between them. Is that because a paper's placement is based on correlation with neighbors, or just random placement?

leonickson 2mo ago
[dead]

abhiram304 2mo ago

[flagged]

Kerbonut 2mo ago

Tailscale Aperture is having trouble doing the OAUth2 connection. Can you support oauth2_dcr or test with Aperture?

leonickson 2mo ago
Yes, DCR is supported (WorkOS exposes the registration endpoint). What error is Aperture throwing? I suspect a metadata-discovery detail.

rahulb0802 2mo ago

I really like the clustering aspect of this. It would be cool if there was a way to articulate some sort of relationship between two adjacent or nearby papers in this tool.

Jahnavi_Aleti 2mo ago

Jumping between arXiv, GitHub, and peer reviews is such a real pain. The “copy page as markdown for your LLM” feature is especially useful.Curious how accurate the entity linking is so far, genes/diseases/etc. That’s usually where these tools get messy.