caravel

6 ms·

1.
by caravel 11mo ago
Hey, creator here. I built this over the past month to try and stay sane while coordinating Claude Code, Codex, and Gemini across projects. Quick feature rundown: - Real-time collaboration with spatial comments/reaction + live cursor
2.
Show HN: Agor → Figma for AI Coding (Open Source)(agor.live)
10 points by caravel 11mo ago|3 comments
3.
by caravel 11mo ago
So I built “Figma for AI coding” to stay sane while coordinating agents across projects, git worktrees, AI sessions and my team. Called it Agor the, "AGent ORchestrator". Built it in a month using Agor to build Agor, plus an army
4.
Promptimize for Test-Driven Prompt Engineering(preset.io)
2 points by caravel 3y ago|1 comments
5.
by caravel 3y ago
A new open source project to support testing prompts at scale by the creator of Airflow & Superset. The toolkit brings many of the ideas from test-driven development to the prompt engineering world, so that people integrating AI in thei
6.
by caravel 4y ago
This is much more straightforward than your JVM-based, huge infra solutions like Flink. For when you need something that is familiar [python] and just works.
7.
by caravel 7y ago
Correction: Prefect is built by one of the Airflow committers. At this time Airflow has ~50 committers. https://whimsy.apache.org/roster/ppmc/airflow
8.
by caravel 8y ago
For context, I used Datastage, Informatica, Ab Initio and SSIS in previous lives an went on to write the first version of Airflow. I developed a taste for pipelines-as-code while working at Facebook using an internal tool that is not open s
9.
by caravel 8y ago
[full disclosure, Airflow committer here] I've never heard of "HP Operation Orchestration", but that looks like a drag and drop enterprise tool from a different Windows-GUI era. Airflow is very different, it's workflows
10.
by caravel 8y ago
[full disclosure, I'm the creator of Airflow] Many environments run tens of thousands of concurrent tasks, and hundreds of thousands of tasks daily. The list of companies using Airflow speaks for itself https://github.com&#x
11.
by caravel 8y ago
Here's the actual talk: https://www.youtube.com/watch?v=23_1WlxGGM4
12.
Superset [open source BI] is joining the Apache Software Foundation(wiki.apache.org)
8 points by caravel 9y ago|1 comments
13.
by caravel 10y ago
[author] Airflow is not a data flow engine, though you can use it to do some of that, but we typically defer on doing data transformations using/coordinating external engines (Spark, Hive, Cascading, Sqoop, PIG, ...). We operate at a h
14.
by caravel 10y ago
[auhtor] Oh cool, I didn't know you guys released your solution yet, I remember demoing Airflow to you guys early on. Looks like it turned out great, congrats on the release!
15.
by caravel 10y ago
At a high level [for Airflow specifically] the scheduler or workflow engine cares most about the tasks and their dependencies, and is somewhat agnostic about the units of work it triggers. It's possible to use a feature called XCom as
16.
by caravel 10y ago
[Airflow author] The task is centric to the workflow engine. It's an important entity, and it's complementary to the data lineage graph (not necessarily a DAG btw). At Airbnb we have another important tool (not open source at the
17.
by caravel 10y ago
[Bias Alert: author of Airflow] can confirm that Airflow allows you to incorporate all of the seven steps, and more as an open platform. At Airbnb Airflow is far from being limited to data engineering. All the batch scheduling goes through
18.
by caravel 10y ago
In a modern data team, Spark is just one of the type of job you may want to orchestrate. Typically as your company gets more tangled in data processing, you'll have many storage and compute engines that you'll have to orchestrate.
19.
by caravel 10y ago
[author] sorry you feel that way. I understand that the section on other similar tools is controversial, especially if you work on one of those. I'm repeating what I hear being very active in this community, and answering the question
20.
by caravel 10y ago
[Airflow author here] in general, and when thinking in terms of best practices, we like to think of a DAG's shape as slowly changing, in a similar way that a database's tables definition is slowly changing. In general, you don&#x2
21.
by caravel 10y ago
I also hear that Uber forked Airflow early on [twice]. I'm curious to hear where they're at at this time.
22.
by caravel 10y ago
[Airflow author here] one of the main differences between Airflow and Luigi is the fact that in Airflow you instantiate operators to create tasks, where with Luigi you derive classes to create tasks. This means it's more natura
23.
by caravel 10y ago
[author of Airflow here] as I wrote in another comment, I'd argue for a programmatic approach to workflows/dataflows as opposed to drag and drop. It turns out that code is a better abstraction for software: https://medi
24.
by caravel 10y ago
I'd argue for something programmatic over drag and drop for reasons described in this other article I wrote: https://medium.freecodecamp.com/the-rise-of-the-data-enginee...
25.
by caravel 10y ago
Many [most] of the companies using Airflow are small-ish scale, maybe less than a half a dozen people writing jobs that need to be scheduled. Airflow will bring clarity even to modest efforts, and will allow to scale the processes as needed
26.
by caravel 10y ago
Maxime reporting for duty here, I'll go through the thread and answer questions / add comments.
27.
by caravel 10y ago
But wait. Isn't S3 "the cloud". Everyone promised the cloud would never go down, ever. It has infinite uptime and reliability. Well good thing I have my backups on [some service that happens to also use S3 as a backend].
28.
by caravel 10y ago
For Superset we use SQLAlchemy and much of the connectivity goes through the DBAPI Python abstraction, and I believe most of the drivers are using native implementations.
29.
by caravel 10y ago
That bug is long fixed. Many companies use Superset against Postgres / Redshift. Superset now ships with a SQL IDE as well.
30.
The Rise of the Data Engineer |-| a Manifesto for the Role(medium.freecodecamp.com)
3 points by caravel 10y ago|0 comments

More ›