8 ms·

Ask HN: What is the best software to visualize a graph with a billion nodes?

Currently I am using GraphViz. But I am not happy with the quality of output (It is writing a postscript file). I want to be able to zoom in and zoom out. Graph has upto 100B nodes and is directed cyclic graph.

oersted 2y ago
It is somewhat old-school, but Gephi is by far the best graph visualization tool I've used that stays robust and usable at such scales (at least ~10M, but possibly a lot more).
- seinecle 2y ago
  And work is underway to help Gephi handle larger graphs: https://gephi.wordpress.com/2024/06/13/gephi-week-2024-peek-from-the-inside/ https://gephi.wordpress.com/2024/06/13/gephi-week-2024-peek-...
deleted 2y ago
[deleted]
rockysharma 2y ago
try dGraph or Aerospike
gkorland 2y ago
Do you just hold this number of node in the database or also need to visualize them all in one view?
- throwaway425933 2y ago
  visualize them in one view so I can understand how data is flowing from input to output
sebstefan 2y ago
Most graphs of social networks done over at /r/dataisbeautiful seem to use Gephi.org and Kumu
shoo 2y ago
what decision / downstream process is going to consume the 1B node graph render? is producing a render really necessary for that decision, or is rendering the graph waste? is there a way you can subsample or simplify or approximate the graph that'd be good enough? in some domains, certain problems that are defined on graphs can be simplified by pre-processing the graph, to reduce the problem to a simpler problem. e.g. maybe trees can be contracted to points, or chains can be replaced with a single edge, or so on. these tricks are sometimes necessary to get scalable solution approaches in industrial applications of optimisation / OR methods to solve problems defined on graphs. a solution recovered on the simplified graph can be "trivially" extended back to the full original graph, given enough post-processing logic. if such graph simplifications make sense for your domain, can you preprocess and simplify your input graph until you hit a fixed point, then visualise the simplified result? (maybe it contracts to 1 node!)
- heresie-dabord 2y ago
  > is producing a render really necessary for that decision, or is rendering the graph waste? Just to be clear, the OP already has a graph. There are nodes and relationships. The graph can be queried for understanding. Rendering the graph is tractable for a small graph or a portion of the graph. Trying to render all the nodes in an enormous graph is almost always an expensive quixotic adventure.
  - hasmanean 2y ago
    Expensive quixotic adventure. Perhaps that is the experience he was after for his billion node graph.
FrustratedMonky 2y ago
Forget a billion. I'm finding even 10's of thousands can be difficult. Just generally, is there a list of visualization products that is broken down by how many nodes they can handle?
- ygra 2y ago
  As someone working on such a visualization product ... it's complex ... and often the wrong question. While you can envision ways of laying out and rendering such large graphs (force-directed layout is frequent, as are hardware-accelerated rendering methods that typically only show nodes with size and color, but little more complex than that), you don't just want to stare at a pretty hairball. Graphs have structure, which the correct layout will emphasize or even make visible. And you want to be able to explore or interact with the data. And there's where this often breaks down. If you're just interested in part of the data, reduce the graph to that part. Makes layout, rendering, and interaction way easier. If you have ways of grouping or clustering the data beforehand, reduce the graph to the clusters and then drill down into them. You might get lucky and your data already has a structure that's well suited for fast layout algorithms and the same structure makes it easy to figure out which part you want to look at more closely. But in my experience that's rare. Most requests for large graphs from customers come from requirements of the software (e.g. “should be able to handle 100k nodes and as many edges at 60 fps with a load time of no more than 2 seconds”) written by someone who pulled more or less reasonable maximum numbers from thin air, or from just looking at the amount of data without really having an idea of how to work with it and just wondering whether all that can somehow be turned into pixels. Dedicating less than a pixel on the screen to each node is very frequently not helpful, even though a visualization product may very well advertise that they can handle it. It may make for pretty pictures, but often not very useful ones. There are a number of posts on the topic, e.g. • https://cambridge-intelligence.com/how-to-fix-hairballs/ https://cambridge-intelligence.com/how-to-fix-hairballs/ • https://www.yworks.com/pages/smooth-visualization-of-big-data-structures https://www.yworks.com/pages/smooth-visualization-of-big-dat...
insomniacity 2y ago
Tacking on a related question - what software should one use to interactively create/update/see a small graph? Thinking specifically about a graph of knowledge, so will be an iterative process. Just looking for anything more than a text editor really!
- oersted 2y ago
  If it's text-heavy I'd recommend Obsidian
  - insomniacity 2y ago
    So the goal would be to eventually move that graph into running code and query it. But it’s never going to be large - easily fit in memory. Obsidian is PKM right? Does it have the idea of labels on the edges?
    - oersted 2y ago
      Right, in that case Obsidian might not be the best choice. I'd look for Knowledge Graph editors then, for RDF or OWL knowledge bases, I don't have any specific recommendation, there are many but most are rather old. Alternatively, go for a graph database like Neo4j, it's primarily designed as a database and not an editor, but it does have a nice UI visualize and change things by hand.
- learn_more 2y ago
  https://schematix.com https://schematix.com
- throwaway425933 2y ago
  dot/vimdot/graphviz promises to do it but I was not able to get it to work yet
mro_name 2y ago
for curiosity - what wisdom do you intend to draw from visualising relations of single gut bacteria? Or is it grains of sand in the sea? How many of them will you zoom into? Maybe clustering may make things feasible.
FL33TW00D 2y ago
It would be a great thing for open source if someone improved the performance of dot/graphviz!
- viraptor 2y ago
  They're not meant for anywhere close to that scale. They deal with text descriptions and rendering full views at the time. That's never going to be usable with more than hundreds of models.
Flam 2y ago
Deck.gl’s PointCloudLayer
wslh 2y ago
https://cytoscape.org/ https://cytoscape.org/ ?
IanCal 2y ago
Datashader is good for rendering large amounts of data, I'd start with that https://datashader.org/ https://datashader.org/
viraptor 2y ago
It really feels like an under defined task. Do you actually need to see those nodes? At that scale, you never want to render 100B of them. Instead you would need some kind of density aggregation when zoomed out and moving to LoD style k-d tree partitioning when zoomed in. That's almost the area of rendering engines like Unreal's Nanite. You can create your own renderer for data like this, but game engines are likely your closest inspiration. Then again, unless you already have x/y coordinates ready, (based on graphviz I'm assuming you don't) even laying out the points will be a very heavy task. (The usual iterative force directed layout would likely take days) But if you were my coworker I'd really press on why do you want the visualisation and if you can get your answers in some other way. And whether you can create aggregates of your data that reduces it to thousands of groups instead. Your data is a minimum of ~800GB if the graph is a single line (position + 64bit value encoding each edge, no labels), so you're not doing anything real-time with it anyway.
- neomantra 2y ago
  Speaking of Nanite, anybody know of a data visualization tools actually implemented with Mesh Shaders? I've dabbled with time series data, not graphs, but it feels lonely.
- quietbritishjim 2y ago
  You're right, but I think that may be what the OP is actually asking for. They talk of "zooming out" but I don't think they mean so they can literally zoom out and see all 100b nodes individually on their screen at once but instead mean that some high level / clustered view is shown to give an overview. That being the case, I think you're suggesting that this high level summarisation happens as a separate preprocessing step (which I agree with FWIW) whereas I think they're imagining it happening dynamically as part of rendering.
- slightwinder 2y ago
  > Do you actually need to see those nodes? Even 8k-screens have not enough pixel to show that many nodes at the same time. So some visual optimization has to happen anyway.
- 2y ago
marcpicaud 2y ago
Sigma.js is pretty good at rendering a ton of nodes and edges. I haven't tried it with a billion nodes though. https://www.sigmajs.org/ https://www.sigmajs.org/
- viraptor 2y ago
  It won't work with hundreds of gigabytes of data. That's not its scope.
macinjosh 2y ago
I haven’t tried that many, but GraphPU was able to render 100s of millions for me in real time. https://github.com/latentcat/graphpu https://github.com/latentcat/graphpu
CuriouslyC 2y ago
You don't. generate a hierarchical clustering of the data, then collapse nodes into groups to get under a data set size threshold at any given view distance. That gives you full interaction and the ability to do mouseover info on groups, while being able to zoom in and interact with individual nodes if you want.
- infecto 2y ago
  This is the way imo. Nobody is consuming 100b nodes in a chart.
  - aeonik 2y ago
    It really depends on what the nodes represent, right? A 1080p monitor has: 1,920 × 1,080 = 2,073,600 pixels Each pixel can display 32-bit color, which equates to: 2^32 = 4,294,967,296 colors So, while each pixel can display one of 4.3 billion colors, the monitor can display combinations of those colors across its 2,073,600 pixels. The total number of possible color combinations on the screen is astronomical. The actual number of possible combinations is: 4,294,967,296^2,073,600
    - tgv 2y ago
      That way of looking at it doesn't make sense. When visualizing a graph, you want to see the connections between the nodes; coloring each node individually almost never makes sense; and the eye cannot distinguish 4B colors.
    - hombre_fatal 2y ago
      You end up bucketing those colors into differences the human eye can see, so you end up with a much smaller domain. You do something similar with 100B data points since you're not literally looking at the relation between individual nodes when all 100B are on screen at once.
    - abeppu 2y ago
      Aside from tgv's correct point that this is implicitly a recipe for something that isn't useful as a visualization, I think even if we were able to distinguish 4B colors and make sense of each pixel -> color assignment ... the math isn't on your side. You responded to a statement about nobody consuming a graph of 100B nodes. Suppose we don't have any concept of edge weight, and an edge is either present or not, but edges are directed, then you have 100B^2 (i, j) pairs for potential edges, each of which is either present or not (i.e. 10^22 edges, each of which is a bit). 4,294,967,296^2,073,600 is very large but 2^(10^22) is much much larger
williamdclt 2y ago
Repeating what others said here: I doubt anyone actually needs to see 1B (or 100B) nodes to make whatever decision they need to make. They probably need to see the X nodes that matter? If you're fully "zoomed out", is seeing 1B individual nodes the most useful representation? Wouldn't some form of clustering be more useful? Same at intermediate levels. D3 has all sorts of graphing tooling and is very powerful. It likely wouldn't handle 1B nodes (even if it did, your browser can't) but it has primitives to build graphs
varjag 2y ago
I remember Tulip could handle pretty huge ones, though no idea if it can manage billions. https://tulip.labri.fr/site/ https://tulip.labri.fr/site/
rcarmo 2y ago
I'd love to see a good solution for this. And it's not just the nodes, it's also the connections between them: https://taoofmac.com/static/graph https://taoofmac.com/static/graph
ARothfusz 2y ago
You could try a hypertree https://en.wikipedia.org/wiki/Hyperbolic_tree https://en.wikipedia.org/wiki/Hyperbolic_tree but that's usually for acyclic data.
OhNoNotAgain_99 2y ago
[dead]
tinsane 2y ago
Dude is casually asking about software to visualize a graph with size comparable to whole internet...
- bee_rider 2y ago
  Maybe it represents the connection between transistors in a chip. Could easily be hundreds of billions of nodes, probably a lot of structure to the edges though.
technologia 2y ago
There was a product I personally liked called graphistry but that isn’t free persay, but its founder is brilliant in this space @lmeyerov
randometc 2y ago
https://www.graphistry.com/ https://www.graphistry.com/ ?
aaron695 2y ago
[dead]
IshKebab 2y ago
100B is going to require something custom, and tbh I'd be surprised if you can get any useful information from that. But try Gephi. It can at least go into the millions of nodes. Not sure about billions.
TZubiri 2y ago
ArangoDB or Neo4j
potatoicecoffee 2y ago
Unformatted csv and you scroll down through it real fast
michaelt 2y ago
You can visualise a graph with 9 billion nodes on https://www.openstreetmap.org https://www.openstreetmap.org :) You could copy their design, if you know how you want to project your nodes into 2D. Essentially dividing the visualisation into a very large number of tiles, generated at 18 different zoom levels, then the 'slippy map' viewer loads the tiles corresponding to the chosen field of view. Then a PostGIS database alongside, letting you run a query to get all the nodes in a given rectangle - such as if you want to find the ID number of a given node.
- vbrandl 2y ago
  I would guess OSM uses optimizations for eucledian graphs, where the path a->c is always shorter than a->b->c. This restriction makes e.g. TSP solvable. But this property does not hold for any generic graph. I don't know if this makes visualisation also easier.
  - michaelt 2y ago
    Technically, if you've got a bumpy dirt track a->c and a freeway a->b->c then the travel time on the latter route can be shorter. Of course, they do get to dodge a major problem: That high-dimensional data is hard to visualise in an understandable way. Everyone knows what a map looks like, nobody knows what a clear visualisation of a set of 100-dimensional vectors looks like.
_flux 2y ago
I'm also looking for a graph viewing tool, but my wishlist is different (not all of them are hard requirements): - Deal with 100k node graphs, preferably larger - Interactive filtering tools, e.g. filtering by node or edge data, transitive closures, highlighting paths matching a condition. Preferably filtering would result in minimally re-layouting the graph. - Does not need an very sophisticated layout algorithms, if hiding or unranking nodes interactively is easy. E.g. centering on a node could layout other nodes using the selected node as the root. - Ability to feed live data externally, add/remove nodes and edges programmatically - Clusters (nodes would tell which clusters they belong in) I'm actually thinking of writing that tool some day, but it would of course be nicer if it already existed ;). I'm thinking applications like studying TLA+ state traces, visualizing messaging graphs or debug data in real time, visualizing the dynamic state of a network. Also if you have tips on applicable Rust crates to help creating that, those are appreciated!
- mhx1138 2y ago
  Neo4j has an interactive graph browser built in.
  - _flux 2y ago
    Thanks! I'll give it a go.
- seinecle 2y ago
  Have you tried Gephi?
  - _flux 2y ago
    I have not! So I gave it a try and it seems quite a capable tool. It doesn't check all my boxes and is more cumbersome to use than I'd wish, e.g. it is able to find shortest path between nodes, but activating that needs finding and entering the node ids manually. However, this still seems the best tool I've ever seen for this purpose and it's also highly general. Thanks!
simpaticoder 2y ago
Hilbert curves (or similar) are often used for graphing billions of nodes[1]. However this will not by default show the relationships between nodes in a graph. Depending on your data you may be able to write a function to map from your edge list to a node index that hints at proximity. Note that visualizations are limited by human perception to ~10000 elements, more usefully 1000 elements. You might try a force directed graph, perhaps a hierarchical variant wherein nodes can contain sub-graphs. Unless you have obvious root nodes, this variant would be interesting in that the user could start from an arbitrary set of nodes, giving different insights depending on their starting point. 1 - An excerpt from "Harder Drive", a rather silly implementation of a unix block device using ping latency with any host that will let him. He visualizes the full ipv4 address space in a hilbert curve at this offset: https://youtu.be/JcJSW7Rprio?si=0AlyMgaZjH7dmh5y&t=363 https://youtu.be/JcJSW7Rprio?si=0AlyMgaZjH7dmh5y&t=363
- zX41ZdbW 2y ago
  Thank you for the link! I've recently created a small service on this topic: https://reversedns.space/ https://reversedns.space/ It visualizes the IPv4 space based on reverse DNS responses.
hhh 2y ago
Graphistry
atemerev 2y ago
For billions of nodes, there are two options: Graphistry (and it might be less than that, 100M is OK), and Pajek, which is weird, but can handle billions of nodes. Neo4j, cytoscape, etc will not work.
simonsarris 2y ago
As someone who's made graphing libraries for over a decade: Are you sure you want to visualize 1 billion nodes? What's the essential thing you're trying to see? Visualizations are great at helping humans parse data, but usually they work best at human scales. A billion nodes is at best looking at clouds, rather than nodes, which can be represented otherwise.
- trueismywork 2y ago
  I have a slightly different use case. I have a dependency graph of tasks with each task having some attached info in form key value pairs. I want to be able to easily visualize the complete graph but then filter out stuff using conditions on attached info. The filtering should hide there nodes/tasks in graph and auto-resize display. What would be a good solution to this?
  - simonsarris 2y ago
    My library (https://gojs.net https://gojs.net) can do that easily. Give it a look, and if you think the price is acceptable for your project, contact us and we can make you a proof-of-concept.
Xcelerate 2y ago
I might be prematurely classifying your question as an instance of the XY problem, but I worked at a company that tried to create something similar — a graph visualization system that could handle 100B nodes as part of our core product and... well... I would caution you not to do so if your purpose is something along those lines. There's almost never a use case where a customer wants to see a gigantic graph. Or researchers. Or family members for that matter. People's brains just don't seem to mesh with giant graphs. Tiny graphs, sure. Sub-graphs that display relevant information, sure. The whole thing? Nah. Unless it's for an art project, in which case giant graphs can be pretty cool looking.
- bunderbunder 2y ago
  I'm reminded of the time back in the aughties when I was asked to help print a ~300,000 page PDF. That's about 30 boxes' worth of paper if you print double-sided. I spent an hour tracing the request back to its source and discovered that they really only wanted some specific pieces of information out of it. I extracted that information from the file and printed maybe 5 pages instead. In moments like these your job is to not be the monkey's paw. Don't just blithely give them what they asked for. Ask more questions to find out what they're actually trying to accomplish, and help them compose a more specific request that's closer to what they actually want.
  - exe34 2y ago
    > Don't just blithely give them what they asked for. Depends on how much they're paying you for it.
    - bunderbunder 2y ago
      Knowing how and when to be consultative is a key soft skill that helps get you access to the higher end of the pay scale. It's how you demonstrate that you're an independent thinker who doesn't need to be micro-managed.
    - pfortuny 2y ago
      Just in case you are not being sarcastic: there is a thing called ethics, which is the basis of human relations.
surrTurr 2y ago
Cytoscape JS[1] with canvas rendering. Probably won't be able to do a billion nodes, but the last time I compared graph rendering libraries it was the best one in terms of performance/customizability. If you need even more performance, there's VivaGraphJS[2], which uses webgl to render. If you want other resources, I also have a GitHub list of Graph-related libraries (visualizations etc.) on GitHub[3]. [1]: https://js.cytoscape.org/ https://js.cytoscape.org/ [2]: https://github.com/anvaka/VivaGraphJS https://github.com/anvaka/VivaGraphJS [3]: https://github.com/stars/AlexW00/lists/graph-stuff https://github.com/stars/AlexW00/lists/graph-stuff
- acomjean 2y ago
  We use cytoscape for some of our genetics tools. It works well. It does tend to “hairball” (technical term) at about 500+ nodes. That’s not the tools fault, just large graphs tend to be difficult. It’s just hard to imagine visualizing a million plus nodes without doing some clustering first.
thomassmith65 2y ago
Your Google Takeout? This sort of thing is why I left /s
bane 2y ago
Visualizing large graphs is a natural desire for people with lots of connected data. But after a fairly small size, there's almost no utility in visualizing graphs. It's much more useful to compute various measures on the graph, and then query the graph using some combination of node/edge values and these computed values. You might subset out the nodes and edges of particular interest if you really want to see them -- or don't visualize at all and just inspect the graph nodes and edges very locally with some kind of tabular data viewer. It used to be thought that visualizing super large graphs would reveal some kind of macro-scale structural insight, but it turns out that the visual structure ends up becoming dominated by the graph layout algorithm and the need to squash often inherently high-dimensional structures into 2 or 3 dimensions. You end up basically seeing patterns in the artifacts of the algorithm instead of any real structure. There's a similar, but unrelated desire to overlay sequenced transaction data (like transportation logs) on a geographical map as a kind of visualization, which also almost never reveals any interesting insights. The better technique is almost always a different abstraction like a sequence diagram with the lanes being aggregated locations. There's a bunch of these kinds of pitfalls in visualization that people who work in the space inevitably end up grinding against for a while before realizing it's pointless or there's a better abstraction. (source: I used to run an infoviz startup for a few years that dealt with this exact topic)
- InGoldAndGreen 2y ago
  The one really helpful use for a massive nodegraph with way too much data? Convincing people that something is complicated. Eg: illustrating to non-technical people that your codebase is a massive mess.
- PaulHoule 2y ago
  I am pretty sour about it and will call out people who post "just another hairball" and act like they've done something special. I think there is a need for a tool that can extract and tell an interesting story based on a subgraph of a huge graph, but that takes thinking unlike hairball plotting, ai image generation and other seductive scourges. I went to an posthumous art show based on this guy https://www.amazon.com/Interlock-Conspiracy-Shadow-Worlds-Lombardi/dp/1619025655 https://www.amazon.com/Interlock-Conspiracy-Shadow-Worlds-Lo... where they showed how he drew 40 drafts with pencil of one of his graphs and went from a senseless hairball to something that seems immediately meaningful. Funny that might have something to do with his mysterous death... Maybe a tool that would help you do that is too dangerous for "them" to let you have!
jarmitage 2y ago
Mosaic is designed for scale https://github.com/uwdata/mosaic https://github.com/uwdata/mosaic https://idl.uw.edu/mosaic/ https://idl.uw.edu/mosaic/
deleted 2y ago
[deleted]
bee_rider 2y ago
Maybe you could turn it into a sparse matrix, hit it with a couple different reorderings, do some matvecs, and see if that gives you any insight into it?
snickerd00dle 2y ago
At that size what you're actually looking for is a game engine with a particle system.
- throwaway425933 2y ago
  Wow. I had not thought of that. While idea makes sense, even MVP for this is going to be daunting
  - samstave 2y ago
    Sent you email with some details on volumetric particle cloud in blender and pythons to snake a harness to that mvp using cuda/other gpu signal tools...
zamalek 2y ago
Try collapsing cycles into single nodes. In my experience, cycles are extremely low entropy. Those cycle nodes can then be explored on separate diagrams/pages. Explore more dimensions that allow you to collapse nodes. You effectively want to turn you graph into a data cube.
david_p 2y ago
As many people already commented, no one actually visualizes graphs of that size at once. Context: I’m the CTO of a GraphViz company, I’ve been doing this for 10+ years. Here are my recommendations: - if you can generate a projection of your graph into millions of nodes, you might be able to get somewhere with Three.js, which is a JS library to generate WebGL graphics. The library is close enough to the metal to allow you to build something large and fast. - if you can get the data below 1M nodes, your best shot is Ogma (spoiler: my company made it). It scales well thanks to WebGL and allows for complex interactions. It can run a graph layout on the GPU in your browser. See https://doc.linkurious.com/ogma/latest/examples/layout-force-gpu-settings.html https://doc.linkurious.com/ogma/latest/examples/layout-force... - If you want to keep your billions of nodes but are OK with not seeing the whole graph at once, my company builds Linkurious. It is an advanced exploration interface for a graph stored in Neo4j (or Amazon Neptune). We believe that local exploration up to 10k nodes on screen is enough, as long as you can run graph queries and full-text search queries against the whole graph with little friction. See https://doc.linkurious.com/user-manual/latest/running-queries/ https://doc.linkurious.com/user-manual/latest/running-querie...
- OutThisLife 2y ago
  Just wanted to say, while I'll never use Ogma, it's really fun to play w/. Performant too.
throwaway425933 2y ago
Thanks to everybody who replied. I will scale my ambitions for now. How can I visualize a one billion node graph. Lets say I want to visualize transitors in a modern AI chip (around 1B nodes). My original use case was to set color on various compinents on a transistor and visualize it. For example, all flops will have one color, all buffers another color, and then I wanted to visualize their distribution on the semiconductor die.
bjourne 2y ago
The amount of information in a graph that big is on the order of 10^21. You can't meaningfully "visualize" it.
withinboredom 2y ago
A couple of years ago, I had a similar issue. I don't have the code any more, but my output to a 3d model converter in a weekend and then threw it into unreal. Then I put on my VR goggles and walked around the graph. It was much easier to deal with in three dimensions instead of two. From there I could write better visualizations. I got laid off before the project was completed, though.
deleted 2y ago
[deleted]
egberts1 2y ago
I was working on a node graph for nftables bison parser. A blog that covers only failures of large SVG viewers having 10,000+ of nodes. https://egbert.net/blog/articles/comparison-svg-viewers-large-file.html https://egbert.net/blog/articles/comparison-svg-viewers-larg... More on https://egbert.net/blog/tags/graphviz.html https://egbert.net/blog/tags/graphviz.html
bpanon 2y ago
Statistics
lori_Mann 2y ago
[flagged]
zdimension 2y ago
I had this question a few years back while working on a social network graph project and trying to render a multi-million node graph. Tried Ogma and it worked quite well but it became too slow when approaching the million. Ended up writing my own renderer in C++ and then Rust. Code here: https://github.com/zdimension/graphrust https://github.com/zdimension/graphrust Tested it up to 5M nodes, renders above 60fps on my laptop's iGPU and on my Pixel 7 Pro. Turns out, drawing lots of points using shaders is fast. Though like everybody else here said you probably don't want to draw that many nodes. Create a lower LoD version of the graph and render it instead
InGoldAndGreen 2y ago
Oh god I ran into this issue! Fewer nodes, but still. I created an HTML page that used vis-network to created a force-directed nodegraph. I'd then just open it up and wait for it to settle. The initial code is here, you should be able to dump it into an LLM to explain: https://github.com/HebeHH/skyrim-alchemy/blob/master/HTMLGraphs/skyrim_ingredients.html https://github.com/HebeHH/skyrim-alchemy/blob/master/HTMLGra... I later used d3 to do pretty much the same thing, but with a much larger graph (still only 100,000 nodes). That was pretty fragile though, so I added an `export to svg` button so you could load the graph, wait for it to settle, and then download the full thing. This kept good quality for zooming in and out. However my nodegraphs were both incredibly messy, with many many connections going everywhere. That meant that I couldn't find a library that could work out how to lay it out properly first time, and needed the force-directed nature to spread them out. For your case of 1 billion nodes, force-directed may not be the way to go.

oersted 2y ago

It is somewhat old-school, but Gephi is by far the best graph visualization tool I've used that stays robust and usable at such scales (at least ~10M, but possibly a lot more).

seinecle 2y ago
And work is underway to help Gephi handle larger graphs: https://gephi.wordpress.com/2024/06/13/gephi-week-2024-peek-from-the-inside/ https://gephi.wordpress.com/2024/06/13/gephi-week-2024-peek-...

deleted 2y ago

[deleted]

rockysharma 2y ago

try dGraph or Aerospike

gkorland 2y ago

Do you just hold this number of node in the database or also need to visualize them all in one view?

throwaway425933 2y ago
visualize them in one view so I can understand how data is flowing from input to output

sebstefan 2y ago

Most graphs of social networks done over at /r/dataisbeautiful seem to use Gephi.org and Kumu

shoo 2y ago

what decision / downstream process is going to consume the 1B node graph render? is producing a render really necessary for that decision, or is rendering the graph waste? is there a way you can subsample or simplify or approximate the graph that'd be good enough? in some domains, certain problems that are defined on graphs can be simplified by pre-processing the graph, to reduce the problem to a simpler problem. e.g. maybe trees can be contracted to points, or chains can be replaced with a single edge, or so on. these tricks are sometimes necessary to get scalable solution approaches in industrial applications of optimisation / OR methods to solve problems defined on graphs. a solution recovered on the simplified graph can be "trivially" extended back to the full original graph, given enough post-processing logic. if such graph simplifications make sense for your domain, can you preprocess and simplify your input graph until you hit a fixed point, then visualise the simplified result? (maybe it contracts to 1 node!)

heresie-dabord 2y ago
> is producing a render really necessary for that decision, or is rendering the graph waste? Just to be clear, the OP already has a graph. There are nodes and relationships. The graph can be queried for understanding. Rendering the graph is tractable for a small graph or a portion of the graph. Trying to render all the nodes in an enormous graph is almost always an expensive quixotic adventure.
- hasmanean 2y ago
  Expensive quixotic adventure. Perhaps that is the experience he was after for his billion node graph.

FrustratedMonky 2y ago

Forget a billion. I'm finding even 10's of thousands can be difficult. Just generally, is there a list of visualization products that is broken down by how many nodes they can handle?

ygra 2y ago
As someone working on such a visualization product ... it's complex ... and often the wrong question. While you can envision ways of laying out and rendering such large graphs (force-directed layout is frequent, as are hardware-accelerated rendering methods that typically only show nodes with size and color, but little more complex than that), you don't just want to stare at a pretty hairball. Graphs have structure, which the correct layout will emphasize or even make visible. And you want to be able to explore or interact with the data. And there's where this often breaks down. If you're just interested in part of the data, reduce the graph to that part. Makes layout, rendering, and interaction way easier. If you have ways of grouping or clustering the data beforehand, reduce the graph to the clusters and then drill down into them. You might get lucky and your data already has a structure that's well suited for fast layout algorithms and the same structure makes it easy to figure out which part you want to look at more closely. But in my experience that's rare. Most requests for large graphs from customers come from requirements of the software (e.g. “should be able to handle 100k nodes and as many edges at 60 fps with a load time of no more than 2 seconds”) written by someone who pulled more or less reasonable maximum numbers from thin air, or from just looking at the amount of data without really having an idea of how to work with it and just wondering whether all that can somehow be turned into pixels. Dedicating less than a pixel on the screen to each node is very frequently not helpful, even though a visualization product may very well advertise that they can handle it. It may make for pretty pictures, but often not very useful ones. There are a number of posts on the topic, e.g. • https://cambridge-intelligence.com/how-to-fix-hairballs/ https://cambridge-intelligence.com/how-to-fix-hairballs/ • https://www.yworks.com/pages/smooth-visualization-of-big-data-structures https://www.yworks.com/pages/smooth-visualization-of-big-dat...

insomniacity 2y ago

Tacking on a related question - what software should one use to interactively create/update/see a small graph? Thinking specifically about a graph of knowledge, so will be an iterative process. Just looking for anything more than a text editor really!

oersted 2y ago
If it's text-heavy I'd recommend Obsidian
- insomniacity 2y ago
  So the goal would be to eventually move that graph into running code and query it. But it’s never going to be large - easily fit in memory. Obsidian is PKM right? Does it have the idea of labels on the edges?
  - oersted 2y ago
    Right, in that case Obsidian might not be the best choice. I'd look for Knowledge Graph editors then, for RDF or OWL knowledge bases, I don't have any specific recommendation, there are many but most are rather old. Alternatively, go for a graph database like Neo4j, it's primarily designed as a database and not an editor, but it does have a nice UI visualize and change things by hand.
learn_more 2y ago
https://schematix.com https://schematix.com
throwaway425933 2y ago
dot/vimdot/graphviz promises to do it but I was not able to get it to work yet

mro_name 2y ago

for curiosity - what wisdom do you intend to draw from visualising relations of single gut bacteria? Or is it grains of sand in the sea? How many of them will you zoom into? Maybe clustering may make things feasible.

FL33TW00D 2y ago

It would be a great thing for open source if someone improved the performance of dot/graphviz!

viraptor 2y ago
They're not meant for anywhere close to that scale. They deal with text descriptions and rendering full views at the time. That's never going to be usable with more than hundreds of models.

Flam 2y ago

Deck.gl’s PointCloudLayer

wslh 2y ago

https://cytoscape.org/ https://cytoscape.org/ ?

IanCal 2y ago

Datashader is good for rendering large amounts of data, I'd start with that https://datashader.org/ https://datashader.org/

viraptor 2y ago

It really feels like an under defined task. Do you actually need to see those nodes? At that scale, you never want to render 100B of them. Instead you would need some kind of density aggregation when zoomed out and moving to LoD style k-d tree partitioning when zoomed in. That's almost the area of rendering engines like Unreal's Nanite. You can create your own renderer for data like this, but game engines are likely your closest inspiration. Then again, unless you already have x/y coordinates ready, (based on graphviz I'm assuming you don't) even laying out the points will be a very heavy task. (The usual iterative force directed layout would likely take days) But if you were my coworker I'd really press on why do you want the visualisation and if you can get your answers in some other way. And whether you can create aggregates of your data that reduces it to thousands of groups instead. Your data is a minimum of ~800GB if the graph is a single line (position + 64bit value encoding each edge, no labels), so you're not doing anything real-time with it anyway.

neomantra 2y ago
Speaking of Nanite, anybody know of a data visualization tools actually implemented with Mesh Shaders? I've dabbled with time series data, not graphs, but it feels lonely.
quietbritishjim 2y ago
You're right, but I think that may be what the OP is actually asking for. They talk of "zooming out" but I don't think they mean so they can literally zoom out and see all 100b nodes individually on their screen at once but instead mean that some high level / clustered view is shown to give an overview. That being the case, I think you're suggesting that this high level summarisation happens as a separate preprocessing step (which I agree with FWIW) whereas I think they're imagining it happening dynamically as part of rendering.
slightwinder 2y ago
> Do you actually need to see those nodes? Even 8k-screens have not enough pixel to show that many nodes at the same time. So some visual optimization has to happen anyway.
2y ago

marcpicaud 2y ago

Sigma.js is pretty good at rendering a ton of nodes and edges. I haven't tried it with a billion nodes though. https://www.sigmajs.org/ https://www.sigmajs.org/

viraptor 2y ago
It won't work with hundreds of gigabytes of data. That's not its scope.

macinjosh 2y ago

I haven’t tried that many, but GraphPU was able to render 100s of millions for me in real time. https://github.com/latentcat/graphpu https://github.com/latentcat/graphpu

CuriouslyC 2y ago

You don't. generate a hierarchical clustering of the data, then collapse nodes into groups to get under a data set size threshold at any given view distance. That gives you full interaction and the ability to do mouseover info on groups, while being able to zoom in and interact with individual nodes if you want.

infecto 2y ago
This is the way imo. Nobody is consuming 100b nodes in a chart.
- aeonik 2y ago
  It really depends on what the nodes represent, right? A 1080p monitor has: 1,920 × 1,080 = 2,073,600 pixels Each pixel can display 32-bit color, which equates to: 2^32 = 4,294,967,296 colors So, while each pixel can display one of 4.3 billion colors, the monitor can display combinations of those colors across its 2,073,600 pixels. The total number of possible color combinations on the screen is astronomical. The actual number of possible combinations is: 4,294,967,296^2,073,600
  - tgv 2y ago
    That way of looking at it doesn't make sense. When visualizing a graph, you want to see the connections between the nodes; coloring each node individually almost never makes sense; and the eye cannot distinguish 4B colors.
  - hombre_fatal 2y ago
    You end up bucketing those colors into differences the human eye can see, so you end up with a much smaller domain. You do something similar with 100B data points since you're not literally looking at the relation between individual nodes when all 100B are on screen at once.
  - abeppu 2y ago
    Aside from tgv's correct point that this is implicitly a recipe for something that isn't useful as a visualization, I think even if we were able to distinguish 4B colors and make sense of each pixel -> color assignment ... the math isn't on your side. You responded to a statement about nobody consuming a graph of 100B nodes. Suppose we don't have any concept of edge weight, and an edge is either present or not, but edges are directed, then you have 100B^2 (i, j) pairs for potential edges, each of which is either present or not (i.e. 10^22 edges, each of which is a bit). 4,294,967,296^2,073,600 is very large but 2^(10^22) is much much larger

williamdclt 2y ago

Repeating what others said here: I doubt anyone actually needs to see 1B (or 100B) nodes to make whatever decision they need to make. They probably need to see the X nodes that matter? If you're fully "zoomed out", is seeing 1B individual nodes the most useful representation? Wouldn't some form of clustering be more useful? Same at intermediate levels. D3 has all sorts of graphing tooling and is very powerful. It likely wouldn't handle 1B nodes (even if it did, your browser can't) but it has primitives to build graphs

varjag 2y ago

I remember Tulip could handle pretty huge ones, though no idea if it can manage billions. https://tulip.labri.fr/site/ https://tulip.labri.fr/site/

rcarmo 2y ago

I'd love to see a good solution for this. And it's not just the nodes, it's also the connections between them: https://taoofmac.com/static/graph https://taoofmac.com/static/graph

ARothfusz 2y ago

You could try a hypertree https://en.wikipedia.org/wiki/Hyperbolic_tree https://en.wikipedia.org/wiki/Hyperbolic_tree but that's usually for acyclic data.

OhNoNotAgain_99 2y ago

[dead]

tinsane 2y ago

Dude is casually asking about software to visualize a graph with size comparable to whole internet...

bee_rider 2y ago
Maybe it represents the connection between transistors in a chip. Could easily be hundreds of billions of nodes, probably a lot of structure to the edges though.

technologia 2y ago

There was a product I personally liked called graphistry but that isn’t free persay, but its founder is brilliant in this space @lmeyerov

randometc 2y ago

https://www.graphistry.com/ https://www.graphistry.com/ ?

aaron695 2y ago

[dead]

IshKebab 2y ago

100B is going to require something custom, and tbh I'd be surprised if you can get any useful information from that. But try Gephi. It can at least go into the millions of nodes. Not sure about billions.

TZubiri 2y ago

ArangoDB or Neo4j

potatoicecoffee 2y ago

Unformatted csv and you scroll down through it real fast

michaelt 2y ago

You can visualise a graph with 9 billion nodes on https://www.openstreetmap.org https://www.openstreetmap.org :) You could copy their design, if you know how you want to project your nodes into 2D. Essentially dividing the visualisation into a very large number of tiles, generated at 18 different zoom levels, then the 'slippy map' viewer loads the tiles corresponding to the chosen field of view. Then a PostGIS database alongside, letting you run a query to get all the nodes in a given rectangle - such as if you want to find the ID number of a given node.

vbrandl 2y ago
I would guess OSM uses optimizations for eucledian graphs, where the path a->c is always shorter than a->b->c. This restriction makes e.g. TSP solvable. But this property does not hold for any generic graph. I don't know if this makes visualisation also easier.
- michaelt 2y ago
  Technically, if you've got a bumpy dirt track a->c and a freeway a->b->c then the travel time on the latter route can be shorter. Of course, they do get to dodge a major problem: That high-dimensional data is hard to visualise in an understandable way. Everyone knows what a map looks like, nobody knows what a clear visualisation of a set of 100-dimensional vectors looks like.

_flux 2y ago

I'm also looking for a graph viewing tool, but my wishlist is different (not all of them are hard requirements): - Deal with 100k node graphs, preferably larger - Interactive filtering tools, e.g. filtering by node or edge data, transitive closures, highlighting paths matching a condition. Preferably filtering would result in minimally re-layouting the graph. - Does not need an very sophisticated layout algorithms, if hiding or unranking nodes interactively is easy. E.g. centering on a node could layout other nodes using the selected node as the root. - Ability to feed live data externally, add/remove nodes and edges programmatically - Clusters (nodes would tell which clusters they belong in) I'm actually thinking of writing that tool some day, but it would of course be nicer if it already existed ;). I'm thinking applications like studying TLA+ state traces, visualizing messaging graphs or debug data in real time, visualizing the dynamic state of a network. Also if you have tips on applicable Rust crates to help creating that, those are appreciated!

mhx1138 2y ago
Neo4j has an interactive graph browser built in.
- _flux 2y ago
  Thanks! I'll give it a go.
seinecle 2y ago
Have you tried Gephi?
- _flux 2y ago
  I have not! So I gave it a try and it seems quite a capable tool. It doesn't check all my boxes and is more cumbersome to use than I'd wish, e.g. it is able to find shortest path between nodes, but activating that needs finding and entering the node ids manually. However, this still seems the best tool I've ever seen for this purpose and it's also highly general. Thanks!

simpaticoder 2y ago

Hilbert curves (or similar) are often used for graphing billions of nodes[1]. However this will not by default show the relationships between nodes in a graph. Depending on your data you may be able to write a function to map from your edge list to a node index that hints at proximity. Note that visualizations are limited by human perception to ~10000 elements, more usefully 1000 elements. You might try a force directed graph, perhaps a hierarchical variant wherein nodes can contain sub-graphs. Unless you have obvious root nodes, this variant would be interesting in that the user could start from an arbitrary set of nodes, giving different insights depending on their starting point. 1 - An excerpt from "Harder Drive", a rather silly implementation of a unix block device using ping latency with any host that will let him. He visualizes the full ipv4 address space in a hilbert curve at this offset: https://youtu.be/JcJSW7Rprio?si=0AlyMgaZjH7dmh5y&t=363 https://youtu.be/JcJSW7Rprio?si=0AlyMgaZjH7dmh5y&t=363

zX41ZdbW 2y ago
Thank you for the link! I've recently created a small service on this topic: https://reversedns.space/ https://reversedns.space/ It visualizes the IPv4 space based on reverse DNS responses.

hhh 2y ago

Graphistry

atemerev 2y ago

For billions of nodes, there are two options: Graphistry (and it might be less than that, 100M is OK), and Pajek, which is weird, but can handle billions of nodes. Neo4j, cytoscape, etc will not work.

simonsarris 2y ago

As someone who's made graphing libraries for over a decade: Are you sure you want to visualize 1 billion nodes? What's the essential thing you're trying to see? Visualizations are great at helping humans parse data, but usually they work best at human scales. A billion nodes is at best looking at clouds, rather than nodes, which can be represented otherwise.

trueismywork 2y ago
I have a slightly different use case. I have a dependency graph of tasks with each task having some attached info in form key value pairs. I want to be able to easily visualize the complete graph but then filter out stuff using conditions on attached info. The filtering should hide there nodes/tasks in graph and auto-resize display. What would be a good solution to this?
- simonsarris 2y ago
  My library (https://gojs.net https://gojs.net) can do that easily. Give it a look, and if you think the price is acceptable for your project, contact us and we can make you a proof-of-concept.

Xcelerate 2y ago

I might be prematurely classifying your question as an instance of the XY problem, but I worked at a company that tried to create something similar — a graph visualization system that could handle 100B nodes as part of our core product and... well... I would caution you not to do so if your purpose is something along those lines. There's almost never a use case where a customer wants to see a gigantic graph. Or researchers. Or family members for that matter. People's brains just don't seem to mesh with giant graphs. Tiny graphs, sure. Sub-graphs that display relevant information, sure. The whole thing? Nah. Unless it's for an art project, in which case giant graphs can be pretty cool looking.

bunderbunder 2y ago
I'm reminded of the time back in the aughties when I was asked to help print a ~300,000 page PDF. That's about 30 boxes' worth of paper if you print double-sided. I spent an hour tracing the request back to its source and discovered that they really only wanted some specific pieces of information out of it. I extracted that information from the file and printed maybe 5 pages instead. In moments like these your job is to not be the monkey's paw. Don't just blithely give them what they asked for. Ask more questions to find out what they're actually trying to accomplish, and help them compose a more specific request that's closer to what they actually want.
- exe34 2y ago
  > Don't just blithely give them what they asked for. Depends on how much they're paying you for it.
  - bunderbunder 2y ago
    Knowing how and when to be consultative is a key soft skill that helps get you access to the higher end of the pay scale. It's how you demonstrate that you're an independent thinker who doesn't need to be micro-managed.
  - pfortuny 2y ago
    Just in case you are not being sarcastic: there is a thing called ethics, which is the basis of human relations.

surrTurr 2y ago

Cytoscape JS[1] with canvas rendering. Probably won't be able to do a billion nodes, but the last time I compared graph rendering libraries it was the best one in terms of performance/customizability. If you need even more performance, there's VivaGraphJS[2], which uses webgl to render. If you want other resources, I also have a GitHub list of Graph-related libraries (visualizations etc.) on GitHub[3]. [1]: https://js.cytoscape.org/ https://js.cytoscape.org/ [2]: https://github.com/anvaka/VivaGraphJS https://github.com/anvaka/VivaGraphJS [3]: https://github.com/stars/AlexW00/lists/graph-stuff https://github.com/stars/AlexW00/lists/graph-stuff

acomjean 2y ago
We use cytoscape for some of our genetics tools. It works well. It does tend to “hairball” (technical term) at about 500+ nodes. That’s not the tools fault, just large graphs tend to be difficult. It’s just hard to imagine visualizing a million plus nodes without doing some clustering first.

thomassmith65 2y ago

Your Google Takeout? This sort of thing is why I left /s

bane 2y ago

Visualizing large graphs is a natural desire for people with lots of connected data. But after a fairly small size, there's almost no utility in visualizing graphs. It's much more useful to compute various measures on the graph, and then query the graph using some combination of node/edge values and these computed values. You might subset out the nodes and edges of particular interest if you really want to see them -- or don't visualize at all and just inspect the graph nodes and edges very locally with some kind of tabular data viewer. It used to be thought that visualizing super large graphs would reveal some kind of macro-scale structural insight, but it turns out that the visual structure ends up becoming dominated by the graph layout algorithm and the need to squash often inherently high-dimensional structures into 2 or 3 dimensions. You end up basically seeing patterns in the artifacts of the algorithm instead of any real structure. There's a similar, but unrelated desire to overlay sequenced transaction data (like transportation logs) on a geographical map as a kind of visualization, which also almost never reveals any interesting insights. The better technique is almost always a different abstraction like a sequence diagram with the lanes being aggregated locations. There's a bunch of these kinds of pitfalls in visualization that people who work in the space inevitably end up grinding against for a while before realizing it's pointless or there's a better abstraction. (source: I used to run an infoviz startup for a few years that dealt with this exact topic)

InGoldAndGreen 2y ago
The one really helpful use for a massive nodegraph with way too much data? Convincing people that something is complicated. Eg: illustrating to non-technical people that your codebase is a massive mess.
PaulHoule 2y ago
I am pretty sour about it and will call out people who post "just another hairball" and act like they've done something special. I think there is a need for a tool that can extract and tell an interesting story based on a subgraph of a huge graph, but that takes thinking unlike hairball plotting, ai image generation and other seductive scourges. I went to an posthumous art show based on this guy https://www.amazon.com/Interlock-Conspiracy-Shadow-Worlds-Lombardi/dp/1619025655 https://www.amazon.com/Interlock-Conspiracy-Shadow-Worlds-Lo... where they showed how he drew 40 drafts with pencil of one of his graphs and went from a senseless hairball to something that seems immediately meaningful. Funny that might have something to do with his mysterous death... Maybe a tool that would help you do that is too dangerous for "them" to let you have!

jarmitage 2y ago

Mosaic is designed for scale https://github.com/uwdata/mosaic https://github.com/uwdata/mosaic https://idl.uw.edu/mosaic/ https://idl.uw.edu/mosaic/

deleted 2y ago

[deleted]

bee_rider 2y ago

Maybe you could turn it into a sparse matrix, hit it with a couple different reorderings, do some matvecs, and see if that gives you any insight into it?

snickerd00dle 2y ago

At that size what you're actually looking for is a game engine with a particle system.

throwaway425933 2y ago
Wow. I had not thought of that. While idea makes sense, even MVP for this is going to be daunting
- samstave 2y ago
  Sent you email with some details on volumetric particle cloud in blender and pythons to snake a harness to that mvp using cuda/other gpu signal tools...

zamalek 2y ago

Try collapsing cycles into single nodes. In my experience, cycles are extremely low entropy. Those cycle nodes can then be explored on separate diagrams/pages. Explore more dimensions that allow you to collapse nodes. You effectively want to turn you graph into a data cube.

david_p 2y ago

As many people already commented, no one actually visualizes graphs of that size at once. Context: I’m the CTO of a GraphViz company, I’ve been doing this for 10+ years. Here are my recommendations: - if you can generate a projection of your graph into millions of nodes, you might be able to get somewhere with Three.js, which is a JS library to generate WebGL graphics. The library is close enough to the metal to allow you to build something large and fast. - if you can get the data below 1M nodes, your best shot is Ogma (spoiler: my company made it). It scales well thanks to WebGL and allows for complex interactions. It can run a graph layout on the GPU in your browser. See https://doc.linkurious.com/ogma/latest/examples/layout-force-gpu-settings.html https://doc.linkurious.com/ogma/latest/examples/layout-force... - If you want to keep your billions of nodes but are OK with not seeing the whole graph at once, my company builds Linkurious. It is an advanced exploration interface for a graph stored in Neo4j (or Amazon Neptune). We believe that local exploration up to 10k nodes on screen is enough, as long as you can run graph queries and full-text search queries against the whole graph with little friction. See https://doc.linkurious.com/user-manual/latest/running-queries/ https://doc.linkurious.com/user-manual/latest/running-querie...

OutThisLife 2y ago
Just wanted to say, while I'll never use Ogma, it's really fun to play w/. Performant too.

throwaway425933 2y ago

Thanks to everybody who replied. I will scale my ambitions for now. How can I visualize a one billion node graph. Lets say I want to visualize transitors in a modern AI chip (around 1B nodes). My original use case was to set color on various compinents on a transistor and visualize it. For example, all flops will have one color, all buffers another color, and then I wanted to visualize their distribution on the semiconductor die.

bjourne 2y ago

The amount of information in a graph that big is on the order of 10^21. You can't meaningfully "visualize" it.

withinboredom 2y ago

A couple of years ago, I had a similar issue. I don't have the code any more, but my output to a 3d model converter in a weekend and then threw it into unreal. Then I put on my VR goggles and walked around the graph. It was much easier to deal with in three dimensions instead of two. From there I could write better visualizations. I got laid off before the project was completed, though.

deleted 2y ago

[deleted]

egberts1 2y ago

I was working on a node graph for nftables bison parser. A blog that covers only failures of large SVG viewers having 10,000+ of nodes. https://egbert.net/blog/articles/comparison-svg-viewers-large-file.html https://egbert.net/blog/articles/comparison-svg-viewers-larg... More on https://egbert.net/blog/tags/graphviz.html https://egbert.net/blog/tags/graphviz.html

bpanon 2y ago

Statistics

lori_Mann 2y ago

[flagged]

zdimension 2y ago

I had this question a few years back while working on a social network graph project and trying to render a multi-million node graph. Tried Ogma and it worked quite well but it became too slow when approaching the million. Ended up writing my own renderer in C++ and then Rust. Code here: https://github.com/zdimension/graphrust https://github.com/zdimension/graphrust Tested it up to 5M nodes, renders above 60fps on my laptop's iGPU and on my Pixel 7 Pro. Turns out, drawing lots of points using shaders is fast. Though like everybody else here said you probably don't want to draw that many nodes. Create a lower LoD version of the graph and render it instead

InGoldAndGreen 2y ago

Oh god I ran into this issue! Fewer nodes, but still. I created an HTML page that used vis-network to created a force-directed nodegraph. I'd then just open it up and wait for it to settle. The initial code is here, you should be able to dump it into an LLM to explain: https://github.com/HebeHH/skyrim-alchemy/blob/master/HTMLGraphs/skyrim_ingredients.html https://github.com/HebeHH/skyrim-alchemy/blob/master/HTMLGra... I later used d3 to do pretty much the same thing, but with a much larger graph (still only 100,000 nodes). That was pretty fragile though, so I added an `export to svg` button so you could load the graph, wait for it to settle, and then download the full thing. This kept good quality for zooming in and out. However my nodegraphs were both incredibly messy, with many many connections going everywhere. That meant that I couldn't find a library that could work out how to lay it out properly first time, and needed the force-directed nature to spread them out. For your case of 1 billion nodes, force-directed may not be the way to go.