frankmcsherry

5 ms·

1.
by frankmcsherry 6y ago
> Sounds like basic memoization and topological sort gets you all the way there? I don't really want to pull rank here, but for the benefit of other readers: 100% nope. I personally find the "make toxic comments to draw folks o
2.
by frankmcsherry 6y ago
I think maybe they were confused by this text (which I agree has nothing to do with Rust itself breaking): > With the release of Apache Arrow 3.0.0 there are many breaking changes in the Rust implementation and as a result it has been ne
3.
by frankmcsherry 6y ago
> If SQL had a way of picking one row from a group, rather than aggregating over it, that would be immensely useful. You can do this with a LATERAL join, if you want to avoid the jankiness of window functions. Lateral joins are just a pr
4.
by frankmcsherry 6y ago
TD lets you write whatever logic you want (it is fairly unopinionated on your logic and state). Differential dataflow plugs in certain logic there, and it does indeed maintain a synopsis of what data have gone past, sufficient to respond to
5.
by frankmcsherry 6y ago
It just comes down to something as simple as: "if I have shown you 1M different things, and now show you one more thing, what do you have to do to tell me whether that one thing is new or not?" If you can keep a hash map of the th
6.
by frankmcsherry 6y ago
There are a few differences, the main one between Spark and timely dataflow is that TD operators can be stateful, and so can respond to new rounds of input data in time proportional to the new input data, rather than that plus accumulated s
7.
by frankmcsherry 6y ago
I think the main distinction is around "interactivity" and how long it takes from typing a query to getting results out. Once you stand up a Flink dataflow, it should move along a brisk clip. But standing up a new dataflow is rela
8.
by frankmcsherry 6y ago
It's a good question, but you'd have to ask them I think. Tamas (from Itemis) and I were in touch for a while, mostly shaking out why DD was out-performing their previous approach, but I haven't heard from him since. My conte
9.
by frankmcsherry 6y ago
Here's my take on this, from a few months back: https://materialize.com/lateral-joins-and-demand-driven-quer...
10.
by frankmcsherry 6y ago
Min and max work using a hierarchical reduction tree, the dataflow equivalent of a priority queue. They will update, under arbitrary changes to the input relation, in time proportional to the number of those changes. > [...] it's ha
11.
by frankmcsherry 6y ago
All queries are incrementally maintained with the property that we do work proportional to the number of records in difference at each intermediate stage of the query plan. That includes those with lateral joins; they are not an exception.
12.
by frankmcsherry 6y ago
I don't think there is anything fundamentally different from an existing database. In all relational databases, some lateral joins can be expensive to compute. In Materialize, those same lateral joins will also be expensive to maintain
13.
by frankmcsherry 6y ago
> I take this to mean that Materialize cannot efficiently maintain a view with lateral joins [...] Well, no this isn't a correct take. Lateral joins introduce what is essentially a correlated subquery, and that can be surprisingly e
14.
by frankmcsherry 6y ago
It's easier to describe the things that cannot be materialized. The only rule at the moment is that you cannot currently maintain queries that use the functions `current_time()`, `now()`, and `mz_logical_timestamp()`. These are quantit
15.
by frankmcsherry 6y ago
Hi! I work at Materialize. I think the right starter take is that Materialize is a deterministic compute engine, one that relies on other infrastructure to act as the source of truth for your data. It can pull data out of your RDBMS's
16.
by frankmcsherry 6y ago
Hi, I work at Materialize. You can read about Vertica's "Live Aggregate Projections" here: https://www.vertica.com/docs/9.2.x/HTML/Content/Authoring/An... In particular, there are imp
17.
by frankmcsherry 6y ago
> Data anonymized with Amnesia are statistically guaranteed that they cannot be linked to the original data. It looks like (from other text on their site) they use variants on k-anonymity. This can prevent re-linking attacks back to
18.
by frankmcsherry 6y ago
Naming things is hard. In this case, the command does not create a new dataflow, it creates a new timely dataflow computation by executing the closure on multiple workers. The computation can then spin zero, one, or many dataflows interacti
19.
by frankmcsherry 6y ago
It's a bit different, right, because that paper presents better algorithms on the same datasets as the work it cites.
20.
Lateral Joins and Demand-Driven Queries(materialize.io)
10 points by frankmcsherry 6y ago|0 comments
21.
by frankmcsherry 6y ago
Yeah, for example I have an open PR in differential dataflow to make some changes to avoid a planned break in the future. https://github.com/TimelyDataflow/differential-dataflow/pull... I think they are doing the
22.
by frankmcsherry 6y ago
> That's not accurate. Fwiw, I recorded at least four different times a Rust release broke timely / differential dataflow, and have seen a few others in other folks' code. Afaict, none of them were soundness related, and w
23.
by frankmcsherry 6y ago
In SQL you would most likely be directed to use `WITH RECURSIVE`, which is something we plan to do, but not yet. It can be a bit gross to use WITH RECURSIVE, because there are often some constraints on the types of queries you can express (
24.
by frankmcsherry 6y ago
By moving up the stack a bit (managing computation, rather than storage) we can provide consistency using techniques other than just using the guarantees provided by the storage itself. This isn't a new observation (Dryad/DryadLIN
25.
by frankmcsherry 6y ago
> That title goes to "no tracking" This is roughly analogous to "abstinence is the best form of birth control". It's not wrong, but it also isn't particularly realistic or helpful. The reality is that people
26.
by frankmcsherry 6y ago
The projects you've mentioned are attempts to address stream processing needs with a SQL-like language. That is fundamentally different from providing incremental view maintenance of actual SQL using streaming techniques (what Materi
27.
by frankmcsherry 6y ago
Hi folks. Frank from Materialize here. The main differences you should expect to see are generality and performance. Generality, in that there are fewer limitations on what you can express. Oracle (and most RDBMSs) build their Incremental V
28.
by frankmcsherry 6y ago
The Tightyl is gorgeous, btw, and made me start to think about what a 3d printer could do for me. For others, check it out at: https://github.com/okke-formsma/dactyl-manuform-tight/
29.
by frankmcsherry 6y ago
I think the median of the results works great in any case (but, you are the expert here). If we find the result median in each of the two inputs, say 'a' and 'b' we have four merges (two forward, two reverse): [0,
30.
by frankmcsherry 6y ago
Thanks for the interesting post! I liked the MergeDouble approach which I hadn’t seen before. Couldn’t that be extended to finding the median of the results using binary search and then doing four concurrent merges, for even more ILP?

More ›