jaychia

9 ms·

1.
by jaychia 5mo ago
Very cool that this kind of work can now be performed at this kind of a price-point. 24 hours for 8M filings on just 12 cores :) Excited for unstructured/multimodal data processing to become increasingly commoditized and abstracted awa
2.
by jaychia 8mo ago
It's fairly surprising to me how naive/early we are still in the techniques that we use here. Anthropic's post on the Claude Agent SDK (formerly Claude Code SDK) talks about how the agent "gathers context", and is f
3.
Knowledge curation (not search) is the AI big data problem(daft.ai)
12 points by jaychia 9mo ago|0 comments
4.
by jaychia 10mo ago
Great article and a timely reminder for many :) Applicable not just for grad school applications, but also to job apps, startups, and relationships. Hang in there y'all, all it takes is for one to work out. Keep working hard, kings &am
5.
by jaychia 1y ago
Check out Daft (www.getdaft.io) - we've been working really hard on our Iceberg support. Supports full reads/writes (including partitioned writes) and our SQL support is also coming along quite well! Also no cluster, no JVM. Just
6.
by jaychia 2y ago
Hey, I'm one of the developers of Daft :) Thanks for the feedback on marketing! Daft is indeed distributed using Ray, but to do so involves Daft being architected very carefully for distributed computing (e.g. using map/reduce par
7.
by jaychia 2y ago
One of the maintainers of Daft here. Just dug through the datachain codebase to understand a little more. I think while both projects have a Dataframe interface , they're very different projects! Datachain seems to operate more on the
8.
by jaychia 2y ago
I work on Daft and we’ve been collaborating with the team at Amazon to make this happen for about a year now! We love Ray, and are excited about the awesome ecosystem of useful + scalable tools that run on it for model training and serving.
9.
by jaychia 2y ago
There’s a lot of interesting work happening in this area (see: XTable). We are building a Python distributed query engine, and share a lot of the same frustrations… in fact until quite recently most of the table formats only had JVM client
10.
by jaychia 3y ago
Interesting. Daft currently does validation on types/names only at runtime. The flow looks like: 1. Construct a dataframe (performs schema inference) 2. Access (now well-typed) columns and operations on those columns in the dataframe,
11.
by jaychia 3y ago
Daft developer here! We actually already have read support. Check out the pyiceberg docs' Daft section: https://py.iceberg.apache.org/api/#daft It's also very easy to use from Daft itself: `daft.read_iceberg(
12.
by jaychia 3y ago
Oh yes good point! We'll be sure to add more details about comparisons with local dataframe libraries such as Pandas/Polars/DuckDB.
13.
by jaychia 3y ago
Hello, Daft developer here! The network indeed becomes the bottleneck. In 2 main ways: 1. Reading data from cloud storage is very expensive. Here’s a blogpost where we talk about some of the optimizations we’ve done in that area: https:&#x
14.
by jaychia 3y ago
Hello! Daft developer here. We are most similar in API to Polars and PySpark. And thanks for the feedback! We’ll add more capabilities for regex, as well as flesh out our documentation for partitioning. Edit: added a new issue for regex sup
15.
by jaychia 3y ago
Hello! Daft developer here. The benchmarks we performed aren’t directly comparable to the benchmarks on TPC-H’s own page because of differences in hardware, storage etc. For hardware, we were using AWS i3.2xlarge machines in a distributed c
16.
by jaychia 3y ago
Hello! Daft developer here - we don’t directly use Polars as an execution engine, but parts of the codebase (e.g. the expressions API) are heavily influenced by Polars code and hence you may see references to Polars in those sections. We do
17.
by jaychia 3y ago
Spent some time diving into the Apache Parquet file format, which was surprisingly complicated and nuanced. There's lots of lore/history in the versioning of the format's various features, and I put together a post to share s
18.
Working with the Apache Parquet file format(blog.getdaft.io)
1 points by jaychia 3y ago|1 comments
19.
by jaychia 3y ago
> Ray backend runner Yes, give it a whirl and let us know what you think! Ray is amazing and has actually gotten a lot better post their 2.0 release :) > Is this based on Apache Arrow? Indeed it is, and thanks for the feedback. We&#
20.
by jaychia 3y ago
We hear you, and thanks for making this visible! As a performance-driven project it’s important for us to understand which operations and use-cases are slowest/buggiest for our users so that we can focus on them. We tried to be very in
21.
by jaychia 3y ago
> Does Daft support Delta table format? Not yet, it’s on our todo list to integrate with the ecosystem of data catalogs (Iceberg/Delta/Hudi etc). Join our Slack/get in touch with us if you’re keen on this though, we’d love
22.
by jaychia 3y ago
> So Daft is a distributed Polars ? We did actually start by using Polars as our underlying execution engine, but eventually transitioned off to our own Rust Table abstraction to better suit our needs (e.g. custom datatypes and kernels).
23.
by jaychia 3y ago
Hi, I'm one of the maintainers of Daft 1. Thanks! We think so too :) 2. Here's my 2c in argument of flat files - Ingestion: ingesting things into a data lake is much easier than writing to a database (all you have to do is drop so
24.
by jaychia 3y ago
Hello! I am one of the maintainers of Daft. Funny enough I just gave a presentation about Daft in London and we all had quite a laugh at the name :D