psfried

9 ms·

1.
by psfried 2y ago
Yes, and this is an important point! This is the reason for our current approach for sqlite derivations. You can absolutely just store all the data in the sqlite database, as long as it actually fits. And there's cases where people act
2.
by psfried 2y ago
I agree completely! We've always talked about this, but we haven't really seen a clear way to package it into a good developer UX. We've got some ideas, though, so maybe one day we'll take a stab at it. For now we'v
3.
by psfried 2y ago
The main benefit isn't necessarily that it's _streaming_ per se, but that it's _incremental_. We typically see people start by just incrementally materializing their data to a destination in more or less the same set tables t
4.
by psfried 2y ago
To my knowledge, nobody's implemented parquet fragment files. But it supports compression of JSONL out of the box. JSON compresses very well, and compression ratios approaching 10/1 are not uncommon. But more to the point, journal
5.
by psfried 2y ago
I don't think it's correct to say that JSONL is any more vulnerable to invalid data than other message framings. There's literally no system out there that can fully protect you from bugs in your own application. But the clie
6.
by psfried 3y ago
Newton at least became quite wealthy. Dunno about the others
7.
by psfried 3y ago
In contrast to jebarker's comment, I actually think it's really interesting that a concept coming from game engine development actually seems quite applicable in some very different domains. We ( https://estuary.dev&#x2F
8.
An off the shelf solution for data products(estuary.dev)
2 points by psfried 3y ago|1 comments
9.
by psfried 3y ago
I feel like "data products" was a great idea, but difficult to implement in practice. There's kind of a paradox where you need a platform in order to host your data products, but any data products that are tied to a specific
10.
End-to-end static type checking of realtime data pipelines(estuary.dev)
4 points by psfried 4y ago|1 comments
11.
by psfried 4y ago
Post explaining why we chose Typescript for realtime data tranformations in Flow, and how it enables end-to-end static type checking of streaming data pipelines.
12.
by psfried 4y ago
> But it is binary, so can’t be viewed or edited with standard tools, which is a pain. I've heard this sentiment expressed multiple times before, and a minor quibble I have with it is that the fact that it's binary has nothing
13.
by psfried 5y ago
That's an interesting statistic about Julia. Makes me curious what the stats would be for Rust.
14.
Don't Just Copy, Materialize(estuary.dev)
2 points by psfried 5y ago|0 comments
15.
by psfried 5y ago
I get the gist of what you're saying, and broadly agree that seasoned programmers tend to develop a strong sense of professional humility. I have to say that I think the analogy goes a bit too far, though. Even very poor programmers ca
16.
by psfried 5y ago
First let me say that I think Timely Dataflow and Materialize are both super cool. The two approaches are quite different, in part because they solve slightly different problems. Or maybe it's more fair to say that they think of the
17.
by psfried 5y ago
IANAE on Flink, especially when it comes to the internals. But I think that the decomposition of computations into distinct map and reduce functions seems to afford a bit more flexibility, since it can be useful to apply reductions separate
18.
by psfried 5y ago
Apart from Google, which has a patent related to their 2004 paper, I don't know how much people are trying to "take credit" for map-reduce. I'm certainly not. But I do think the approach of running map-reduce continuousl
19.
by psfried 5y ago
I agree with this. As soon as the MapReduce paper came out, people were criticizing it for a lack of novelty, claiming that so-and-so has been using these same techniques for years. And of course those critics are still around saying the sa
20.
by psfried 5y ago
Versioning is indeed an issue, but that's the case for anything with long-lived state. Our current rely on JSON schemas, TypeScript, and built-in testing support to help ensure compatibility. Those things actually help quite a bit in p
21.
MapReduce is making a comeback(estuary.dev)
73 points by psfried 5y ago|38 comments
22.
by psfried 5y ago
Another possible explaination is simply that people have gotten better at designing programming languages. Or that newer languages are better adapted to solving the problems we now want to solve. Which really shouldn't be too hard to s