5 ms·
What if we extended this idea beyond one dataset to all discrete news events and entities: people, organizations, places. Just like here you could get a timeli
by pickpuck 10mo ago
What if we extended this idea beyond one dataset to all discrete news events and entities: people, organizations, places.
Just like here you could get a timeline of key events, a graph of connected entities, links to original documents.
Newsrooms might already do this internally idk.
This code might work as a foundation. I love that it's RDF.
- pjc50 10mo agoSomeone did one for (a small subset of) UK media. People were furious. https://brokenbottleboy.substack.com/p/mapped-out https://brokenbottleboy.substack.com/p/mapped-out
- j-pb 10mo agoIf it's RDF it won't work as the foundation.
- axus 10mo agoOne wonders what the US government agencies use.
- abnercoimbre 10mo agoI think you meant one shudders. And yeah, Snowden made it clear there's orders of magnitude more data than this graph explorer for them to sift through.
- PaulHoule 10mo agoIsn’t that what Palantir’s product is?
- sswaner 10mo agoPretty much, at least at the semantic layer. https://publish.obsidian.md/followtheidea/Content/AI/Ontology+Palantir+-+notes https://publish.obsidian.md/followtheidea/Content/AI/Ontolog...
- cjohnson318 10mo agoThey probably use Excel, maybe Microsoft Access.
- ToucanLoucan 10mo agoMicrosoft Access form that connects via IIS to an Excel spreadsheet acting as a database. Also the server it's running on is sitting on a wooden table.
- cjohnson318 10mo agoBro you can't just leak operational secrets on the world wide information highway like this.
- fancy_pantser 10mo agoSoftware like i2 Analyst's Notebook.
- dboreham 10mo agoInternet search engines have their origins in government projects fwiw. They had search engines before Alta Vista, used for searching data sets that pre-date the internet, and some of the people involved in those went to work on the original commercial search engines.
- arthurcolle 10mo agoProbably not particularly useful but GCHQ & NSA both have neat graph related repos UK: https://github.com/gchq/Gaffer https://github.com/gchq/Gaffer US: https://github.com/NationalSecurityAgency/lemongraph https://github.com/NationalSecurityAgency/lemongraph
- VikingCoder 10mo agoSci-Fi Author: In my book I invented the Torment Nexus as a cautionary tale Tech Company: At long last, we have created the Torment Nexus from classic sci-fi novel Don't Create The Torment Nexus
- throwaway290 10mo ago...and of course it's in RDF!
- darth_aardvark 10mo agoPalantir, arguably the closest thing to Torment Nexus Inc. IRL, literally builds a product that does this.
- FanaHOVA 10mo agoOne co trying: https://www.system.com https://www.system.com
- jandrewrogers 10mo agoThis has been attempted many times. They all fail the same way. These general data models start to become useful and interesting at around a trillion edges, give or take an order of magnitude. A mature graph model would be at least a few orders of magnitude larger, even if you aggressively curated what went into it. This is a simple consequence of the cardinality of the different kinds of entities that are included in most useful models. No system described in open source can get anywhere close to even the base case of a trillion edges. They will suffer serious scaling and performance issues long before they get to that point. It is a famously non-trivial computer science problem and much of the serious R&D was not done in public historically. This is why you only see toy or narrowly focused graph data models instead of a giant graph of All The Things. It would be cool to have something like this but that entails some hardcore deep tech R&D.
- babelfish 10mo agoI don't have any experience on graph modeling, but it seems like Neo4j should be able to support 1 trillion edges, based on this (admittedly marketing) post of theirs? https://neo4j.com/press-releases/neo4j-scales-trillion-plus-relationship-graph/ https://neo4j.com/press-releases/neo4j-scales-trillion-plus-...
- jandrewrogers 10mo agoThe graph database market has a deserved reputation for carefully crafting scaling claims that are so narrowly qualified as to be inapplicable to anything real. If you aren't deep into the tech you'll likely miss it in the press releases. It is an industry-wide problem, I'm not trying to single out Neo4j here. Using this press release as an example, if you pay attention to the details you'll notice that this graph has an anomalously low degree. That is, the graph is very weakly connected, lots of nodes and barely any edges. Typical graph data models have much higher connectivity than this. For example, the classic Graph500 benchmark uses an average degree of 16 to measure scale-out performance. So why did they nerf the graph connectivity? One of the most fundamental challenges in scaling graphs is optimally cutting them into shards. Unlike most data models, no matter how you cut up the graph some edges will always span multiple shards, which becomes a nasty consistency problem in scale-out systems. Scaling this becomes exponentially harder the more highly connected the graph. So basically, they defined away the problem that makes graphs difficult to scale. They used a graph so weakly connected that they could kinda sorta make it work on a thousand(!) machines even though it is not representative of most real-world graph data models.
- johongo 10mo agoEmil Eifrem (founder of Neo4j) has a talk about them doing this with the Panama papers
- afavour 10mo agoThe New York Times has an API that lets you query “tags” or “topics” and the articles associated with them: https://developer.nytimes.com/docs/semantic-api-product/1/overview https://developer.nytimes.com/docs/semantic-api-product/1/ov... The Guardian has similar: https://open-platform.theguardian.com/documentation/tag https://open-platform.theguardian.com/documentation/tag Either or both could be an interesting starting point for something like that. I tried to find something for the BBC and was surprised they didn’t have anything. I would have figured public media would have been a great resource for this.
- ggm 10mo agoGiven 6 degrees is rooted in reality, this means we can draw causal graphs from anyone (bad) to anyone (we don't like) and then invent specious reasons why it means "it's all connected, man" That said, some networks of shorter paths than 6 are interesting. Right now, there's a 1:1 direct path from these documents to a bunch of people with an interest in confounding what evidentiary value they have in justice processes. That's more interesting to me, than what the documents say right now.
- Centigonal 10mo agoCheck out GDELT! https://www.gdeltproject.org/ https://www.gdeltproject.org/