jacquesnadeau

5 ms·

1.
by jacquesnadeau 8mo ago
Apple container is more akin to a replacement for docker or colima (although patterned more like Kata containers where each container is a separate vm as opposed to a bunch of containers in a single vm). It's a promising project (and n
2.
by jacquesnadeau 8mo ago
cool to see ctenv. definitely a similar vibe. thanks for sharing! will look at more closely. Interesting to see how you incorporated some dockerfile patterns. devcontainer feature-esque. I'm curious to know if you are using it for the
3.
by jacquesnadeau 8mo ago
I'm one of the creators of shai. Thanks for the callout! Interesting to see the work on Yolobox and in this space generally. The pattern we've seen as agent use grows is being thoughtful about what different agents get access to.
4.
by jacquesnadeau 9y ago
Arrow supports a union type for heterogeneous columns (we use it for random json in Dremio) and a 128-bit decimal.
5.
by jacquesnadeau 9y ago
PrestoDB is similar to Impala, Hive and other SQL Engines. Each is designed to do distributed SQL processing. Dremio does embed an OSS distributed SQL processing engine (Sabot, built natively on Arrow) as well but we see that as only a mean
6.
by jacquesnadeau 9y ago
Arrow is all about in-memory, not long-term persistence. Systems can write it to disk but it is the one in-memory representation to rule them all, not storage/disk. Disk has its own requirements and challenges outside the scope of Ar
7.
by jacquesnadeau 9y ago
For those level of operations, the Arrow library itself will probably have something fairly soon.
8.
by jacquesnadeau 9y ago
It's more than an implementation detail because we're also targeting interoperability between multiple separate technologies. One of the key things that the article didn't fully cover is that Arrow serves two purposes: high p
9.
by jacquesnadeau 9y ago
Dremio is focused on a combination of data access, acceleration and a self-service analyst experience (Tableau/Qlik one-click integration, data curation and data management). We also invest heavily in our pushdowns. For example, we inv
10.
by jacquesnadeau 9y ago
People are already using Arrow for high performance GPU processing. FPGA is possible but I'm not sure if anyone is actually doing this today.
11.
by jacquesnadeau 9y ago
One quick note to make on this. Kudu is a storage implementation, (similar to Parquet in some ways). Arrow isn't about persistence and is actually built to be complementary to both Kudu and Parquet. Also note: Kudu is a distributed pro
12.
by jacquesnadeau 9y ago
The Arrow project itself is a set of libraries. One of the things we'll do is try to add more algorithms over time to it so if you want say, a fast arrow sort or arrow predicate application. Full SQL is always far more complex and I ca
13.
by jacquesnadeau 9y ago
You nailed it. The most exciting part about all of this is being able to move between a "data science" context and a "database" context (and back again) without pain or penalty.
14.
by jacquesnadeau 9y ago
Possible, yes. Performant or pleasant? Maybe not :)
15.
by jacquesnadeau 9y ago
We built Dremio (github.com/dremio/dremio-oss, apache licensed) entirely on top of Apache Arrow specifically for the purposes of creating a high speed analytical capabilities including MOLAP like work as well as other forms of cac
16.
by jacquesnadeau 9y ago
I'm also a developer on Arrow ( https://github.com/jacques-n/ ), similar to WesM. It is always rewarding (and also sometimes challenging) to hear how people understand or value something you're working on. I th