7 ms·

How NASA Is Using Graph Technology and LLMs to Build a People Knowledge Graph

107 points by lexmo67 1y ago

rkwz 1y ago
Any idea which LLM they're using?
mistrial9 1y ago
NASA announced LLMs in early days (years ago) - it seemed like they wanted to understand their own document libraries! What else could be inferred here? mass layoffs plus "people substitutes" ? is there a more diplomatic way to see this?
- karamanolev 1y ago
  The only connection between "they wanted to understand their own document libraries" and "mass layoffs" is potentially "increased efficiency leads to needing less people for the same job". If there's anything else, please let me know. And if it's that, then are you suggesting to not implement a certain technological efficiency tool in order to keep (now clearly redundant) jobs? That has never worked long-term in the history of mankind, AFAIK.
behnamoh 1y ago
> What made you choose memgraph? ... And then Memgraph showed me the cost. That kind of sold me for time for us to be able to do that. It's an ad post about memgraph.
- ctxc 1y ago
  Yes, domain is memgraph and it seems to be a marketing case study.
dpflan 1y ago
As an alternative to a pure graph db (e.g. here, memgraph), has anyone here used Apache's AGE graph-database extension for Postgresql? For making a knowledge graph that can live alongside SQL?
- dgllghr 1y ago
  I believe AGE has unfortunately been defunded: https://github.com/apache/age/discussions/2150 https://github.com/apache/age/discussions/2150 It’s a shame because it seemed like being able to query data across multiple paradigms would be really useful
  - UltraSane 1y ago
    My dream databases is Neo4j style relationships and MongoDB style documents.
    - dgllghr 1y ago
      Databases like that do exist. There is ArangoDB for example. I have no idea if it’s any good
    - grounder 1y ago
      Check out SurrealDB. It might be exactly what you're looking for. https://surrealdb.com/ https://surrealdb.com/
demaga 1y ago
> 27K nodes and 230K edges This is such an overkill for that kind of data. Even if they do plan to "scale up significantly", I doubt that they'll actually experience any benefit of graph db.
- mmooss 1y ago
  Why do you say that?
jerryseff 1y ago
Memgraph is laughably expensive - I honestly wonder what anyone actually uses it for outside of companies that just don't care about infra spend.
- mbuda 1y ago
  DISCLAIMER: The co-founder and CTO of Memgraph here. To add more context, Memgraph Enterprise pricing is explained under https://memgraph.com/pricing https://memgraph.com/pricing: "Starting at $25,000 per year for 16 GB, Memgraph has an all-inclusive, simple pricing model that scales with your workload without restrictions. No charge for compute. No charge for replicas. No charge for algorithms. No Surprises.". In addition, Memgraph Community is free (standard BSL license, which turns into Apache2 4 years after release date, https://github.com/memgraph/memgraph/blob/master/licenses/BSL.txt https://github.com/memgraph/memgraph/blob/master/licenses/BS...), and it has many features that are usually considered enterprise (users, replication, not a single degradation in performance or scale, etc.). Please elaborate more about why the pricing seems expensive, or put it into the infra-cost perspective :pray:
  - smarx007 1y ago
    I think on this site anything that's more expensive than free is considered expensive. Countless arguments have been had on Oracle vs Postgres, including lock-in. I think lock-in is more important to consider than license cost. To be fair, it is quite nice for the pricing to be transparent. And I think it's somewhat competitive w.r.t. Stardog, for example. The community version is less restricted than Ontotext, for example.
    - kendallgclark 1y ago
      Not really competitive with Stardog given our leading LLM integration with Voicebox. 85% pass@1 to exit POV with new customer.
- smarx007 1y ago
  If you want a production-grade graph DBMS, you don't have that many OSS options that are reliable and well-supported. In the relational space, it took OSS options like Postgres many decades (and somehow paid-for person-years) to get to a place where enterprises seriously consider migrating off Oracle to it.
timewizard 1y ago
> It’s [sql] just not built for the complex relationships that exist in a massive organizations like NASA. This is an absurd claim. > Extracted Skills from Team Resumes > Extracted Skills > Subject Matter Experts Finder Question: designed to identify employees with expertise in specific domains or mission-critical capabilities. I can't think of anything that screams "incompetent management" more than this. So, to find a subject matter expert, you're going to "extract skills" and "extract resumes" to answer abstract questions about your staff... without ever once.. just _talking_ to your staff? What a cold and bizarre future these people think we want to live in. Meanwhile can we use technology to improve the level of connectivity I have and experience as an employee? Can you please stop asking LLMs to "extract" things about me into goofy automated pipelines? If you want skilled workers then you need to demonstrate skilled management. This is all the exact opposite of that.
- deleted 1y ago
  [deleted]
- mbuda 1y ago
  This is like saying: "Here we have a rocket, but let's keep trying to go to the moon by bike." xD What's wrong with attempting to better understand a given organization using LLMs or any other tech? Ofc, great managers will try as hard as possible to talk face to face as much as possible.
  - timewizard 1y ago
    > This is like saying: I highly doubt the difference between current staff management and adding this thin layer is equivalent to difference between a bike and a rocket. It's more like saying "we get to the moon just fine, but if we strap this extra booster on, we will get there 2% faster than before but with all kinds of additional risks to the payload!" > What's wrong with attempting to better understand a given organization You can alienate your employees and lose your skill base as a result. I'd like to be evaluated based on upon my work and dedication, not what some LLM thinks it sees in my resume. I've worked for my current company for 17 years. My resume contains none of that work or any skills gained in that time. I also like to take on new challenges and learn new skills. The LLMs "extractions" cannot see this or attend to it. > Ofc, great managers will try as hard as possible to talk face to face as much as possible. That's not the problem being discussed here. The question is "can we use technology to make better organizational decisions particularly when it comes to the efficient use of human resources." If I have a bad boss, I'm going to quit, and you'll never even have this opportunity. If I have a good boss, and you interfere with his decisions using LLM driven logic, I'm going to quit, and you're never going to get the benefit of that labor anyways.
deleted 1y ago
[deleted]
inerte 1y ago
I know it’s a marketing case study, but: > Ever wondered how NASA identifies its top experts, forms high-performing teams, and plans for the skills of tomorrow? Here’s another resource on that https://appel.nasa.gov/2010/02/18/aa_2-7_f_nasa_teams-html/ https://appel.nasa.gov/2010/02/18/aa_2-7_f_nasa_teams-html/ the book “How NASA Builds Teams: Mission Critical Soft Skills for Scientists, Engineers, and Project Teams”
cebert 1y ago
I think I found a place Dodge can save some money. Memgraph pricing is ridiculous.
- patcon 1y ago
  Even paying a college grad to babysit a server costs more than their yearly rate. I assume you're speaking as someone who loves to host everything for themselves, but the logic is surely different in enterprise/government, no?
  - cebert 1y ago
    It depends on your usage models, but if you compare it to AWS Neptune, the pricing seems quite high. I doubt NASA is running queries 24x7 for this use case so other options could be less expensive.
jandrewrogers 1y ago
> The current graph has about 27K nodes and 230K edges That is tiny even by historical standards. I was expecting there to be some type of technology here. Why is this interesting?
smarx007 1y ago
> "To make sure everyone understands that, I prefer label property graphs over RDF." I have two major issues with virtually all graph DBMSs that are not RDF/SPARQL-based: 1) They do not allow structure-preserving querying. That is, I query a graph and want the results to be a smaller graph. This is trivial in SQL, you just 'SELECT * FROM x WHERE ...' and the result set you get is tabular just like the table x. In SPARQL, there are a CONSTRUCT/DESCRIBE queries that do just that - give you the results as a graph. 2) They don't use any (internationally recognized) standard to represent graph data. RDF is the only such format known to me (ignore all the semantic web stuff associated with it and just consider the format). 230k edges is peanuts for a graph db. It's like when the number of rows times columns in your SQL DB is 230k. NASA could (should?) have just used Oxigraph, RDF4J, or Jena. Stardog and Ontotext are the paid options. However, it is quite nice to see more interest in graph-based DBMSs in general! > “Which employees have cross-disciplinary expertise in AI/ML?” Regarding the study itself, I did not understand who is the target user of this. I would rather be more interested in the Lessons Learned 2.0 study (I understand it was attempted once before [1]). I don't think the study at hand would be able to correctly answer questions about expertise. On the technical side, as far as I understand, the cosine similarity was computed per triplet? In that case, I could see how pgvector could be used for this. Relevance expansion is the only thing in the article that made me think that it would be cool if it works well. But I could see how in a combo of a regular RDF DBMS + pgvector, one could first do a cosine similarity query via pgvector and then compute an (S)CBD [2] of the subject (the from node) of the triplet. [1]: https://youtu.be/QEBVoultYJg?t=1653 https://youtu.be/QEBVoultYJg?t=1653 [2]: https://patterns.dataincubator.org/book/bounded-description.html https://patterns.dataincubator.org/book/bounded-description....
- UltraSane 1y ago
  "They do not allow structure-preserving querying. That is, I query a graph and want the results to be a smaller graph." I'm not sure what you mean by this. The result of a query in neo4j is a set of nodes with specified relations linking them. It is much more flexible than the way SQL can only return a single table.
  - smarx007 1y ago
gitroom 1y ago
Man, love seeing pushback on automated skill matchingsometimes feels like tech folks keep inventing new tools just to dodge actual conversations. Ever wonder if all this automation just makes things colder instead of smarter?
- rage4774 1y ago
  And slower, if we‘re honest, since they never solve issues they intend to solve 100% and human attention is still needed (which is good)
citizenpaul 1y ago
My experience with tools like this is that they have only one single outcome. Piling work onto the most talented or desperate(ie need money or visa) people until they leave the org/company. Eventually leading to total skill erosion and a very low average skill/productivity across the company as people leave or hide their abilities. Why? because there is never a reward attached. Oh you want to make me the AI resource for the agency but not remove former duties or increase my pay? Ummmm no thanks. Also things tend to happen in waves ie "AI" so everyone needs a lot from a very few people at the same time. No one ever asks how those people can be empowered. Just how can we put the screws to them so they work harder. HR and Mgmt can f-off with their "skill resource bank" or whatever nonsense they call it this year. My skills are what I was hired for on the job description. If you want to discuss a new position or higher pay for different skills I'm very happy to talk about how I can work with the org to make that happen. Thats never the case though.
dcreater 1y ago
Talks extensively about the details of the thing. But doesn't actually show the thing. That's AI hypecycle signal for probably bullshit/defective thing.
thumbsup-_- 1y ago
Seems like a very simple use-case given that it will be barely used at scale. A few thousand employee entries and read qps a few 10s? What’s so special about it to post
kendallgclark 1y ago
The use case at NASA isn’t even new. We built this precise thing in 2008. All standards-based. See https://www.w3.org/2001/sw/sweo/public/UseCases/Nasa/ https://www.w3.org/2001/sw/sweo/public/UseCases/Nasa/ for a public case study. This work led to Stardog. Which is used extensively in NASA today— https://gpdisonline.com/wp-content/uploads/2019/09/StardogNASA-Schain-AcceleratingModelBasedSys-MBSE-Open.pdf https://gpdisonline.com/wp-content/uploads/2019/09/StardogNA... https://www.informationweek.com/machine-learning-ai/stardog-ceo-on-fueling-nasa-s-ai-mission-and-enabling-data-relationships https://www.informationweek.com/machine-learning-ai/stardog-...
neets 1y ago
Yall know that Apache Age is a thing to run Cypher in Postgres right?
PeterStuer 1y ago
HP Tried this over 20 years ago. It stranded in HR and union disputes. My own take is this will just be gamed to the max by ladder climbers looking out for number 1.
truetaurus 1y ago
Interesting, i am just going to plug something I just built around this concept: https://skillriskaudit.com/ https://skillriskaudit.com/ Would love for some feedback!

107 points by lexmo67 1y ago

rkwz 1y ago

Any idea which LLM they're using?

NASA announced LLMs in early days (years ago) - it seemed like they wanted to understand their own document libraries! What else could be inferred here? mass layoffs plus "people substitutes" ? is there a more diplomatic way to see this?

karamanolev 1y ago
The only connection between "they wanted to understand their own document libraries" and "mass layoffs" is potentially "increased efficiency leads to needing less people for the same job". If there's anything else, please let me know. And if it's that, then are you suggesting to not implement a certain technological efficiency tool in order to keep (now clearly redundant) jobs? That has never worked long-term in the history of mankind, AFAIK.

behnamoh 1y ago

> What made you choose memgraph? ... And then Memgraph showed me the cost. That kind of sold me for time for us to be able to do that. It's an ad post about memgraph.

ctxc 1y ago
Yes, domain is memgraph and it seems to be a marketing case study.

dpflan 1y ago

As an alternative to a pure graph db (e.g. here, memgraph), has anyone here used Apache's AGE graph-database extension for Postgresql? For making a knowledge graph that can live alongside SQL?

dgllghr 1y ago
I believe AGE has unfortunately been defunded: https://github.com/apache/age/discussions/2150 https://github.com/apache/age/discussions/2150 It’s a shame because it seemed like being able to query data across multiple paradigms would be really useful
- UltraSane 1y ago
  My dream databases is Neo4j style relationships and MongoDB style documents.
  - dgllghr 1y ago
    Databases like that do exist. There is ArangoDB for example. I have no idea if it’s any good
  - grounder 1y ago
    Check out SurrealDB. It might be exactly what you're looking for. https://surrealdb.com/ https://surrealdb.com/

demaga 1y ago

> 27K nodes and 230K edges This is such an overkill for that kind of data. Even if they do plan to "scale up significantly", I doubt that they'll actually experience any benefit of graph db.

mmooss 1y ago
Why do you say that?

jerryseff 1y ago

Memgraph is laughably expensive - I honestly wonder what anyone actually uses it for outside of companies that just don't care about infra spend.

mbuda 1y ago
DISCLAIMER: The co-founder and CTO of Memgraph here. To add more context, Memgraph Enterprise pricing is explained under https://memgraph.com/pricing https://memgraph.com/pricing: "Starting at $25,000 per year for 16 GB, Memgraph has an all-inclusive, simple pricing model that scales with your workload without restrictions. No charge for compute. No charge for replicas. No charge for algorithms. No Surprises.". In addition, Memgraph Community is free (standard BSL license, which turns into Apache2 4 years after release date, https://github.com/memgraph/memgraph/blob/master/licenses/BSL.txt https://github.com/memgraph/memgraph/blob/master/licenses/BS...), and it has many features that are usually considered enterprise (users, replication, not a single degradation in performance or scale, etc.). Please elaborate more about why the pricing seems expensive, or put it into the infra-cost perspective :pray:
- smarx007 1y ago
  I think on this site anything that's more expensive than free is considered expensive. Countless arguments have been had on Oracle vs Postgres, including lock-in. I think lock-in is more important to consider than license cost. To be fair, it is quite nice for the pricing to be transparent. And I think it's somewhat competitive w.r.t. Stardog, for example. The community version is less restricted than Ontotext, for example.
  - kendallgclark 1y ago
    Not really competitive with Stardog given our leading LLM integration with Voicebox. 85% pass@1 to exit POV with new customer.
smarx007 1y ago
If you want a production-grade graph DBMS, you don't have that many OSS options that are reliable and well-supported. In the relational space, it took OSS options like Postgres many decades (and somehow paid-for person-years) to get to a place where enterprises seriously consider migrating off Oracle to it.

timewizard 1y ago

> It’s [sql] just not built for the complex relationships that exist in a massive organizations like NASA. This is an absurd claim. > Extracted Skills from Team Resumes > Extracted Skills > Subject Matter Experts Finder Question: designed to identify employees with expertise in specific domains or mission-critical capabilities. I can't think of anything that screams "incompetent management" more than this. So, to find a subject matter expert, you're going to "extract skills" and "extract resumes" to answer abstract questions about your staff... without ever once.. just _talking_ to your staff? What a cold and bizarre future these people think we want to live in. Meanwhile can we use technology to improve the level of connectivity I have and experience as an employee? Can you please stop asking LLMs to "extract" things about me into goofy automated pipelines? If you want skilled workers then you need to demonstrate skilled management. This is all the exact opposite of that.

deleted 1y ago
[deleted]
mbuda 1y ago
This is like saying: "Here we have a rocket, but let's keep trying to go to the moon by bike." xD What's wrong with attempting to better understand a given organization using LLMs or any other tech? Ofc, great managers will try as hard as possible to talk face to face as much as possible.
- timewizard 1y ago
  > This is like saying: I highly doubt the difference between current staff management and adding this thin layer is equivalent to difference between a bike and a rocket. It's more like saying "we get to the moon just fine, but if we strap this extra booster on, we will get there 2% faster than before but with all kinds of additional risks to the payload!" > What's wrong with attempting to better understand a given organization You can alienate your employees and lose your skill base as a result. I'd like to be evaluated based on upon my work and dedication, not what some LLM thinks it sees in my resume. I've worked for my current company for 17 years. My resume contains none of that work or any skills gained in that time. I also like to take on new challenges and learn new skills. The LLMs "extractions" cannot see this or attend to it. > Ofc, great managers will try as hard as possible to talk face to face as much as possible. That's not the problem being discussed here. The question is "can we use technology to make better organizational decisions particularly when it comes to the efficient use of human resources." If I have a bad boss, I'm going to quit, and you'll never even have this opportunity. If I have a good boss, and you interfere with his decisions using LLM driven logic, I'm going to quit, and you're never going to get the benefit of that labor anyways.

deleted 1y ago

[deleted]

inerte 1y ago

I know it’s a marketing case study, but: > Ever wondered how NASA identifies its top experts, forms high-performing teams, and plans for the skills of tomorrow? Here’s another resource on that https://appel.nasa.gov/2010/02/18/aa_2-7_f_nasa_teams-html/ https://appel.nasa.gov/2010/02/18/aa_2-7_f_nasa_teams-html/ the book “How NASA Builds Teams: Mission Critical Soft Skills for Scientists, Engineers, and Project Teams”

cebert 1y ago

I think I found a place Dodge can save some money. Memgraph pricing is ridiculous.

patcon 1y ago
Even paying a college grad to babysit a server costs more than their yearly rate. I assume you're speaking as someone who loves to host everything for themselves, but the logic is surely different in enterprise/government, no?
- cebert 1y ago
  It depends on your usage models, but if you compare it to AWS Neptune, the pricing seems quite high. I doubt NASA is running queries 24x7 for this use case so other options could be less expensive.

jandrewrogers 1y ago

> The current graph has about 27K nodes and 230K edges That is tiny even by historical standards. I was expecting there to be some type of technology here. Why is this interesting?

smarx007 1y ago

> "To make sure everyone understands that, I prefer label property graphs over RDF." I have two major issues with virtually all graph DBMSs that are not RDF/SPARQL-based: 1) They do not allow structure-preserving querying. That is, I query a graph and want the results to be a smaller graph. This is trivial in SQL, you just 'SELECT * FROM x WHERE ...' and the result set you get is tabular just like the table x. In SPARQL, there are a CONSTRUCT/DESCRIBE queries that do just that - give you the results as a graph. 2) They don't use any (internationally recognized) standard to represent graph data. RDF is the only such format known to me (ignore all the semantic web stuff associated with it and just consider the format). 230k edges is peanuts for a graph db. It's like when the number of rows times columns in your SQL DB is 230k. NASA could (should?) have just used Oxigraph, RDF4J, or Jena. Stardog and Ontotext are the paid options. However, it is quite nice to see more interest in graph-based DBMSs in general! > “Which employees have cross-disciplinary expertise in AI/ML?” Regarding the study itself, I did not understand who is the target user of this. I would rather be more interested in the Lessons Learned 2.0 study (I understand it was attempted once before [1]). I don't think the study at hand would be able to correctly answer questions about expertise. On the technical side, as far as I understand, the cosine similarity was computed per triplet? In that case, I could see how pgvector could be used for this. Relevance expansion is the only thing in the article that made me think that it would be cool if it works well. But I could see how in a combo of a regular RDF DBMS + pgvector, one could first do a cosine similarity query via pgvector and then compute an (S)CBD [2] of the subject (the from node) of the triplet. [1]: https://youtu.be/QEBVoultYJg?t=1653 https://youtu.be/QEBVoultYJg?t=1653 [2]: https://patterns.dataincubator.org/book/bounded-description.html https://patterns.dataincubator.org/book/bounded-description....

UltraSane 1y ago
"They do not allow structure-preserving querying. That is, I query a graph and want the results to be a smaller graph." I'm not sure what you mean by this. The result of a query in neo4j is a set of nodes with specified relations linking them. It is much more flexible than the way SQL can only return a single table.
- smarx007 1y ago

gitroom 1y ago

Man, love seeing pushback on automated skill matchingsometimes feels like tech folks keep inventing new tools just to dodge actual conversations. Ever wonder if all this automation just makes things colder instead of smarter?

rage4774 1y ago
And slower, if we‘re honest, since they never solve issues they intend to solve 100% and human attention is still needed (which is good)

citizenpaul 1y ago

My experience with tools like this is that they have only one single outcome. Piling work onto the most talented or desperate(ie need money or visa) people until they leave the org/company. Eventually leading to total skill erosion and a very low average skill/productivity across the company as people leave or hide their abilities. Why? because there is never a reward attached. Oh you want to make me the AI resource for the agency but not remove former duties or increase my pay? Ummmm no thanks. Also things tend to happen in waves ie "AI" so everyone needs a lot from a very few people at the same time. No one ever asks how those people can be empowered. Just how can we put the screws to them so they work harder. HR and Mgmt can f-off with their "skill resource bank" or whatever nonsense they call it this year. My skills are what I was hired for on the job description. If you want to discuss a new position or higher pay for different skills I'm very happy to talk about how I can work with the org to make that happen. Thats never the case though.

dcreater 1y ago

Talks extensively about the details of the thing. But doesn't actually show the thing. That's AI hypecycle signal for probably bullshit/defective thing.

thumbsup-_- 1y ago

Seems like a very simple use-case given that it will be barely used at scale. A few thousand employee entries and read qps a few 10s? What’s so special about it to post

kendallgclark 1y ago

The use case at NASA isn’t even new. We built this precise thing in 2008. All standards-based. See https://www.w3.org/2001/sw/sweo/public/UseCases/Nasa/ https://www.w3.org/2001/sw/sweo/public/UseCases/Nasa/ for a public case study. This work led to Stardog. Which is used extensively in NASA today— https://gpdisonline.com/wp-content/uploads/2019/09/StardogNASA-Schain-AcceleratingModelBasedSys-MBSE-Open.pdf https://gpdisonline.com/wp-content/uploads/2019/09/StardogNA... https://www.informationweek.com/machine-learning-ai/stardog-ceo-on-fueling-nasa-s-ai-mission-and-enabling-data-relationships https://www.informationweek.com/machine-learning-ai/stardog-...

neets 1y ago

Yall know that Apache Age is a thing to run Cypher in Postgres right?

PeterStuer 1y ago

HP Tried this over 20 years ago. It stranded in HR and union disputes. My own take is this will just be gamed to the max by ladder climbers looking out for number 1.

truetaurus 1y ago

Interesting, i am just going to plug something I just built around this concept: https://skillriskaudit.com/ https://skillriskaudit.com/ Would love for some feedback!