6 ms·
Apache AGE, a PostgreSQL extension with graph database functionality
- atemerev 4y ago"Apache AGE is currently being developed for the PostgreSQL 12 release" Well sorry, we have PostgreSQL 15 already.
- gutbasokchok 4y agoand I believe the more recent the version is, the less stable it is. Sure there will be additional nice features, but older versions are still in progress. And yeah, if you kept reading, AGE sounds like it will support more versions in the future.
- feike 4y agoIf you keep reading, the words following your quote make it clear that PG15 should be supported: "and will support PostgreSQL 13 and all the future releases of PostgreSQL."
- metadat 4y agoWhat subset of the openCypher dialect specification will Apache Age support? Every implementation is all over the place and completely non-portable. And neo4j performance leaves much to be desired. My personal go-to is RedisGraph paired with RedisInsight for the instant visualizations. It just feels "right" and, while not perfect, is overall intuitive.
- bavell 4y agoI've been following RedisGraph with great interest since I first heard about it. Fell in love with cypher but not a big fan of neo4j. Most recently I've been playing with EdgeDB (not cypher-based) which scratches my graphDB itch pretty well but it's still a little too early for me to consider adopting it in a serious project. I'm glad there seems to be continued interest in graphDBs, I think there's a lot of potential in that space and I'm eagerly awaiting a clear winner to emerge.
- jhoechtl 4y agoWe might see a clear winner in query language, not product. What is the clear winner in relational databases? SQL certainly but what product?
- flymetothemoon 4y agoThe Apache AGE is inspired by AgensGraph, a multimodel database fork of PostgreSQL. So it uses AgensGraph dialect.
- bayesian_horse 4y agoAGE is an extension, not a fork. You can have relational tables and graphs in the same database, probably you can also use both in the same query.
- flymetothemoon 4y agoThat's right, AGE is designed to use both queries either separately or together. SQL (PostgreSQL), openCypher (AGE)
- faizshah 4y agoAnyone tried this? How does the performance compare to neo4j and RedisGraph? I’m about to give RedisGraph a try and I guess I will try this one a go as well.
- ysko75 4y agoAGE 1.1 should perform better or at least similar to Neo4j. Not sure about RedisGraph.
- tluyben2 4y agoDid you try dgraph? For our use cases it won over neo4j. Didn’t try redisgraph.
- tofuahdude 4y agoWe benchmarked all 3; redisgraph was fastest by far.
- bryanrasmussen 4y agoOk, but on what metrics? I don't mean I disbelieve, I would just like to know that it was fastest on the things I might care about.
- anentropic 4y agoit's an in-memory graph, I'd kind of expect it to be faster than the others
- tluyben2 4y agoThanks, will try it soon.
- bayesian_horse 4y agoDownside is it's harder to keep all data in memory and Redis has less sophisticated query facilities. Not sure how AGE stacks up in that regard, especially interesting would be joining graph data and non-graph data.
- canadiantim 4y agoI’ve been waiting to see Apache AGE on HN. Looks amazing, thanks for all the great work!
- gorlomi 4y agoHow are graph edges and nodes exposed to the Postgresql type system?
- flymetothemoon 4y agoThere is a documentation for AGE that has some basic examples on how cypher and sql can be mixed. Regarding nodes and edges: they are sub-types to a type called Agtype. More details are in the documentation. https://age.apache.org/age-manual/master/index.html https://age.apache.org/age-manual/master/index.html
- CptNibblesworth 4y agoAGE uses Agtype l, which is a superset of JsonB for its uses, the primary addition is the edge and vertex type.
- dang 4y agoRelated: Apache Age: A Graph Extension for PostgreSQL - https://news.ycombinator.com/item?id=26345755 https://news.ycombinator.com/item?id=26345755 - March 2021 (45 comments) Apache AGE: PostgreSQL-based graph database - https://news.ycombinator.com/item?id=26309560 https://news.ycombinator.com/item?id=26309560 - March 2021 (11 comments)
- canadiantim 4y agoCould it be efficient to use Apache AGE for e.g. retrieving all comments on an article? Currently I’m using materialized paths to efficiency return all commments but would be keen to know if AGE can help query comments for an article more powerfully.
- andrewstuart2 4y agoHave you tried recursive CTEs with a simple id, parent_id etc schema? These should perform very well if those columns are in an index. Afaik this is pretty much the canonical way to store recursive comment trees. Or any kind of DAG.
- CptNibblesworth 4y agoAGE can handle that and recursive ctes can as well, but AGE has mechanisms to handle cyclic graphs as well.
- mrslave 4y agoThis comment gave me a flashback to Celko's SQL for Smarties. I believe the updated books are split off into a few smaller books? But the section/book on trees in a relational database helped me greatly once in a galaxy far far away.
- ramraj07 4y agoWhy not a trigger that maintains this in a simpler query in a separate table? Sounds more performant to me! Recursive CTEs sounds like something you would do if your total comment count in the db is not in the six figures or something. What does HN do?
- akshayshah 4y agoAs long as comments are a tree, there’s only one path from the root (the post) to an individual comment. How would a recursive CTE perform better than a prefix scan on an indexed string column? Storing a pointer to each node’s parent or using sorted sets seems like it would make the parent poster’s query slower. Those approaches would make it easier to reparent comments, though, and they’d support arbitrarily deep trees (whereas the materialized path implementations I’ve seen limit path length).
- ysko75 4y agoApache AGE Discord https://discord.com/invite/NMsBs9X8Ss https://discord.com/invite/NMsBs9X8Ss
- mradek 4y agoInteresting. What are some good extensions for pg? I have only used UUID and postgis.
- somebee 4y agoThe ltree extension is fantastic if you have data like comments or any other hierarchical structure.
- jschrf 4y agoTimescaleDB
- hans_castorp 4y ago> I have only used UUID and postgis. If you used "uuid-ossp" to get uuid_generate_v4(), then this is no longer necessary since Postgres 13 as there is now a built-in gen_random_uuid() https://www.postgresql.org/docs/current/functions-uuid.html https://www.postgresql.org/docs/current/functions-uuid.html
- mradek 4y agoOh wow TIL thank you
- ellisv 4y agopg_stat_statements
- Dowwie 4y agoaws_s3: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_PostgreSQL.S3Import.html https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_...
- rdevsrex 4y agoWell, I wouldn't call it interesting, but I like the citext extension for case insensitive comparison.
- sandGorgon 4y agohas anyone here worked on graph neural networks ? basically creating embeddings for node based on their edge connectivity (or reachability) and using that for neural networks ? how do you do this at scale ?its generally a NP hard problem, but wondering whether something like AGE helps. not sure how Google, etc or even someone on fraud detection does this at scale
- ArnoVW 4y agoYou subsample. One package I used made N 'random walks' for each node. The random walks are written out as 'sentences', where the node id's are words. That results in a huge text file, that you then embed as if it were a normal text. The result is a normal 'word embedding' where the words are in reality the node id's. Works like a charm. Highly scalable. https://github.com/dwslab/jRDF2Vec https://github.com/dwslab/jRDF2Vec
- sandGorgon 4y agoreally ? so u keep subsampling as the data becomes larger and larger. instead of ...well...throwing more hardware that seems to be easier and easier these days. P.S. not trolling. im genuinely wondering if there is a better way to split the problem heuristically
- ArnoVW 4y agoAll I'm saying is that you don't take into account all paths for each node. Just for ex 100 random walks starting at each node. And that results in an embedding that is 'good enough'. Of course it is better to throw more hardware at the issue. But at a certain point the added value of being more precise or adding more hardware becomes moot, because you gain 0.1% That is what I meant by 'it scales'. You can solve 'reasonably complex issues' with 'reasonably cheap hardware'
- fulafel 4y agoThis is written in C. I wonder how common it is to write PG extensions in safer langugaes and what would be the most suitable. I'm somewhat wary of using nontrivial C extensions, having seen so many of them sometimes seg fault the backend (eg PostGIS). There seem to be PG backend crashes described in this projects issues as well.
- jpnc 4y agoThere's https://github.com/tcdi/pgx https://github.com/tcdi/pgx for writing extensions in rust.
- nine_k 4y agoPostgres itself is written in C. I suppose its every internal interface is in C. I wonder how many unsafe sections would an extension written in Rust have to have to use these interfaces. I wish something like Lua + LuaJIT could be used to write such extensions; at least it's memory-safe. OTOH mapping these C interfaces to Lua structures, and making them work with GC may happen to be non-trivial.
- fulafel 4y agoPG ships with Lua support: https://www.postgresql.org/docs/current/external-pl.html https://www.postgresql.org/docs/current/external-pl.html (Also Python, Javascript, and Java) I don't know specifics about the API coverage. It seems this extension mostly just implements new SQL visible functions and data types, which should be doable from those languages as well. Composite types might have to be defined as PG records (or json) instead of C level new PG object types.
- enugu 4y agoWish Postgres had an optional data type (tagged union), Does someone know an extension which implements that?
- gyre007 4y agoThis is a great project, but last time I checked it was lacking a lot of CYPHER features and wasn’t moving very fast forward. But I’m hoping it will catch up to the point it will become useful.
- robertlagrant 4y agoCypher as in the Neo4J query language?
- bayesian_horse 4y agoThere is now an OpenCypher specification and AGE seems to strive towards supporting most of it.
- Dowwie 4y agoI create a DAG using recursive sql. I assume that saving data in a graph and querying the graph with a native graph language would be faster. Has anyone benchmarked performance differences between the two?
- bayesian_horse 4y agoDepends on the use case. Recursive SQL can be good enough, maybe even faster, for certain use cases. The problem isn't so much the query language but the indexing. Graph engines index the nodes and edges in a particular way so that traversal is fast. Most examples of Recursive SQL I've seen will only involve nodes on exactly one Table an with exactly one kind of a relationship/edge (for example a tree with "parent" edges). Graph DBs allow you to relate multiple different types of nodes using multiple kinds of edges. The edges can have queryable attributes like an intermediary table in a many-to-many relationship. And all of that is still indexed efficiently.
- bayesian_horse 4y agoAlways bet on PostgreSQL! I hope AGE matures a bit in the future. There are lots of use cases for Graph Databases. One I'm interested in is bitemporality. It's easy to use ltree or CTE for tree-like structures. But what if you want to move nodes in the graph at certain times? Like a device being scheduled to be in different rooms across time. And also the history of those schedules. In a graph database you can label edges with temporal attributes and then query for a view of the graph at a certain point in time and in a certain history state by filtering the edges.
- stevesimmons 4y agoThat's a really interesting idea. Can you recommend any good references for bitemporality in graph dbs?
- bayesian_horse 4y agoTheres XTDB
- stevesimmons 4y agoThank you! XTDB Core2 looks very interesting. I've worked a lot with Arrow, which makes it extra appealing.
- bayesian_horse 4y agoI'm hesitant with databases like these because they are less battle-proven. I normally want my database to be as boring as possible.
- Kalanos 4y agoDoes this construct edge tables (many-to-many) for every relationship behind the scenes? if so, can attributes be added to the edges?
- robertlagrant 4y agoI'd hope so! You can do that in relational DBs.
- mark_l_watson 4y agoI might try running this with Docker just to try it out, but probably this is the type of project to watch and wait for maturity. I am a big fan of graph databases. Professionally I have used RDF data stores with SPARQL queries and Google’s Knowledge Graph with a pattern matching query mode. I play around with Neo4J, but no one has paid me to use it yet. I think it very likely that in a year or two AGE will get better Cypher query language support and other changes, and should be a wonderful platform for combining relational and graph data stores.
- twaway23 4y agoI'm considering using a graph database for a SaaS product. If I used Apache AGE, I would probably have a "graph" for each customer to partition the data. Are there any downsides or limitations to having thousands of separate graphs? From the documentation it seems that each graph will use a separate "namespace" in Postgres. Are there any performance costs of switching namespaces for each query? Or do you recommend that we use a single graph with a label per customer? This option seems like it could open up some security issues if some queries forget to add this label. By using a separate graph per customer, the query will need to have a valid graph name for a customer to return any data. If it is filtered by a label, you can easily forget to add it and think everything is OK because it actually returns results.