8 ms·
Prismatic's "Graph" at Strange Loop
- a-priori 14y agoVery interesting, needs more examples though. It seems like this fits a similar purpose to Nathan Marz' Storm framework. Could you do a bit of a compare/contrast thing between Graph and Storm?
- programnature 14y agoGraph is not distributed, at least not in the way that Storm is. Notice the post doesn't talk about assigning individual components to individual machines. AFAICT its intended to run on a beefy box, and then horizontally scaled.
- tlrobinson 14y agoYou could probably write a higher order function (like "observe-graph" in their example) to distribute the work across multiple machines. Maybe even map it onto a Storm cluster automatically.
- w01fe 14y agoThanks! There are some more complex examples in the slides, I just didn't want to make the blog post any longer. The main similarity between this and Nathan Marz's Storm framework is that they both rely on a declarative expression of the structure of the computation graph. However, beyond that there are many key differences. Storm is a graph computation framework, which compiles your specification into a distributed real-time computation pipeline. In contrast, Graph is just a library for expressing composition structure, but says nothing about execution strategy. In principle (with a lot more code and some more annotations), one could compile Graphs into distributed real-time topologies like Storm. For now we've been using Graph in-process for real-time processing. But because Graph is so simple and close to the language, it's very easy to apply to new situations and build new abstractions on top of. For example, we also find Graph useful for expressing the composition structure of our production services, which are built up from many components.
- jwr 14y agoBut, from what I understand, if I structure my software using Graph, it should be much easier to run it using Storm, even without automated tooling. That way Graph becomes a stepping stone: structure your software, and if you need Storm's parallelism/scalibility, define a topology using the same functions.
- aleyan 14y agoNice. Assuming that the computation of each node is memoized, can we 'dirty' a single node, so that during the next computation only the needed parts of the graph are recomputed? This would make multiple computation of models where only a few parameters are changing much more efficient. Also, has this at all been inspired by SecDB's powerful graph oriented features?
- w01fe 14y agoWe haven't had the need for that application yet, but it's certainly possible -- that's why declarative specifications are great :) I haven't heard of SecDB, but I'll definitely check it out -- thanks!
- gajomi 14y agoThis is definitely a use case I am interested in. I have been playing around with computer algebra in Clojure for awhile where it is critical to keep track of these dependencies (I have a hacky implementation which relies on metadata so that the resulting data structure is out of site). But of course, when you actually want to evaluate something, this is when you need to get sophisticated. There can be multiple paths in the graph to arrive at the same node and you have to decide. With memoization at nodes being an important consideration. Anyway, I would be very happy to see an open source version of this! P.S. As I understand it all the graphs are dags. In my line of work (which involves doing stuff at the interface of probability and statistics) there are quite a few recursive functions that are difficult to unroll explicitly, or are otherwise undesirable to unroll for the purposes of readability. Can we expect to see circular dependencies at some point. Well, I know, the halting problem and all, but still...
- jpadvo 14y agoThis sounds very useful. I'm having trouble understanding the nature of the system, though, because I don't really understand lisp / clojure. Is this a pattern that could be applied in any language with first class functions, or does it require a lisp to be practical?
- w01fe 14y agoSure! This particular way of expressing declarative structure may be specific to Clojure, but the basic ideas should be useful in any (especially dynamic) language with first-class functions. The first idea is to take a complex system and express the set of components and their relationships declaratively, rather than procedurally. This is an old idea, and makes it possible to reason about the system, mock out components for testing, map over the components for monitoring, and more. The second idea is simplicity. By making this declarative language as simple as possible, it becomes trivially easy to do all the things I just mentioned using the existing tools in the language, without needing to write complex library functions to support each use case. For another example of a declarative system for composition you could look at 'react' in JS, which can be used in similar ways to Graph (and also supports things like async composition): https://github.com/jeffbski/react https://github.com/jeffbski/react
- wonderzombie 14y agoreact looks interesting, although man, the stringly-typed bit sorta reinforces a lot of the Lisp quotes about being able to modify the language, etc. (Disclaimer: I've only just started studying Clojure in earnest over the last month or two. No zealot like a convert, eh?)
- jpadvo 14y agoThanks! That makes a lot of sense, and react looks like a really interesting project. I'll have to go back and reread the article with an eye for applying its ideas in Javascript / Coffeescript.
- grayrest 14y agoYou can do it in any language with first class functions and reflection. Here's a stupid simple js implementation of their example: https://gist.github.com/3874826 https://gist.github.com/3874826 Edit: Loaded the page and got interrupted by lunch. Shows me for not refreshing.
- deleted 14y ago[deleted]
- jessepollak 14y agoJust a heads up, it looks like something is up with your CSS: http://cl.ly/image/241X1C2D3O35 http://cl.ly/image/241X1C2D3O35 I'm on 10.8 with the latest version of safari. In other news, this looks pretty cool.
- janaboruta 14y agoAre you using a Retina Macbook?
- renderkid 14y agoI have the same issue and i'm using a Retina MBP.
- janaboruta 14y agoThanks guys. We will look into it.
- salimmadjd 14y agoSame issue on iPad 3 that has retina display
- marbemac 14y agoYup looks like a Retina problem.
- abp 14y agoI had this same problem on my Android phones Chrome browser and went to my laptop for reading.
- pnathan 14y agoIf I understand the nature of this system, it's essentially a higher-order function composition tool. Correct? Nit, defnk? what about def-kw-func or something? not wholly readable as defnk.
- w01fe 14y agoYep, that's correct. Point taken about the names, thanks for the feedback. Since Clojure already uses defn and fn, defnk and fnk seemed natural. But maybe defn-keyword and fn-keyword would be better.
- pnathan 14y agoI really like this idea. Do you have a Common Lisp port of this, out of curiosity? If not I might have to see what I can come up with. :-) I like to have fully expanded names, ala Common Lisp. My editor will autocomplete those on demand after I use a name once, so it's not a hindrance to reading or writing. My goal for identifiers is that I can read without consulting my internal lookup table.
- w01fe 14y agoThanks! Nope, no common lisp port -- and we haven't released the Clojure yet, but plan to soon. It's only 150 lines of code or so should be easy to port or rewrite from scratch.
- dkincaid 14y agoThere was a defnk at one time in one of the Clojure libraries (I think it was even created by Rich himself), but it disappeared at some point. Nathan Marz created a defnk macro in Storm that he uses for the same thing. Maybe it's time for Clojure to bring defnk back if it offers some advantages over the :keys destructuring.
- salimmadjd 14y agoI love reading prismatic's blog. You have some really smart people there. However, it seems like you're trying to solve everything from a purist view and I think your product is suffering as a result. I have made the same suggestion consistently that would make me use it more without any reply. Basically, I've stopped using the product as the UX does not match the reason I would want to use the product.
- janaboruta 14y agoI'm sorry about that. If you're still interested in giving us UX feedback send us an email feedback@getprismatic.com. I'll personally respond to it. Please understand that we're still a small team and we get a lot of emails on a daily basis. :-)
- dmix 14y agoI agree with the author, the UX is definitely under-developed and I've also given many feedback msgs. We only say this because we care :)
- tensor 14y agoI'm curious what your complaints are, do you mind sharing? Especially because you invoke "purist view", which is rather confusing talking about UX, a very subjetive area. For myself, I feel that the main feed could use a lot more whitespace and have less information about each story. For example, I wouldn't mind being able to toggle off the social quotes completely and perhaps even the images (images are nice for some sources, but not others).
- emeraldd 14y agoGraph reminds me of working with Spring (from what I remember of that). The concepts sound very similar to Spring's auto-wiring.
- DennisP 14y agoHappy to see you're opensourcing. Are you planning to do the same with Flop and Optimize?
- w01fe 14y agoFlop is definitely on the list (with about 10 other libraries), but I'm not sure about our optimization code. We plan to start a discussion of what we should prioritize soon, so stay tuned.
- tel 14y agoThis feels like lazy evaluation bolted on to Clojure (cool!). There's further benefit since you get some dataflow abstraction—which then reminds me of the Par monad. http://community.haskell.org/~simonmar/par-tutorial.pdf http://community.haskell.org/~simonmar/par-tutorial.pdf
- mattdeboard 14y agoThere's no need to "bolt on" lazy evaluation since (future) and (delay) are parts of the core language (not to mention lazy seqs)
- moomin 14y agoWell, future, delay and lazy sequences are bolt ons to an eagerly evaluated language. I'm not convinced the laziness is Haskell is worth it, tbh, but its true that you really wouldn't need this project.
- tel 14y agoUsually these kinds of demos are the same as the ones touting the virtues of Haskell's laziness. The visualization is really nice, though, along with the possible paralleization. I think the Par monad is a more interesting comparison, honestly. http://hackage.haskell.org/packages/archive/monad-par/0.3/doc/html/Control-Monad-Par.html http://hackage.haskell.org/packages/archive/monad-par/0.3/do...
- blaines 14y agoIs it just me or is the prismatic blog text being cut off on the side? https://twitter.com/blaines/status/256545302926995456/photo/1/large https://twitter.com/blaines/status/256545302926995456/photo/...
- moomin 14y agoHappens on the iPhone to everyone, I think.
- blaines 14y agoAppears this may be a retina mbp thing...
- xtrahotsauce 14y agohappens to me on an iPad as well
- moomin 14y agoThis is great, but I wish they'd take it one stage further. At the moment, the first example they give (monolithic let) is pretty idiomatic and readable. The transformation from one to the other is pretty automatable. So for most cases, why not just put a macro on top? e.g. defgraph-asfn Storm requires some manual intervention because you need direct control over how things are distributed.
- rafaelferreira 14y agoPretty cool project, like a functional reinterpretation of dependency injection frameworks. The basic idea behind dependency injection is to make all dependencies for each unit explicit and externally supplied. DI frameworks then build upon the pattern offering a declarative DSL to declare dependencies and wiring. They also offer a selection of alternatives to build dependent objects (e.g. build a fresh one when needed or supply a global instance for every dependee) and a smidgen of generic functionality via impersonation (e.g. monitoring, logging, timing, transaction acquisition/release). Interesting that untyped functional programming and macros allow for a lighter expression of the same basic idea. Also interesting that maps of functions are a fundamental part of the solution. There are definitions of object-orientation that amount to basically records-of-lambdas. I've long thought that there is a lot to gain by using this basic abstraction as it naturally provides seams for testing/observability/auditing/etc.