kumare3

6 ms·

1.
Flyte 2 is GA: durable distributed AI workflows using regular Python(union.ai)
18 points by kumare3 2mo ago|2 comments
2.
by kumare3 3y ago
May I recommend looking into flyte.org, it is open source kubernetes native "orchestration" style tool, but essentially and infrastructure component that is geared to making your ML Engineers and Data scientists more productive.
3.
by kumare3 4y ago
Thank you for sharing. If you do not have platform engineers look at union.ai. They offer a manager version of Flyte
4.
by kumare3 4y ago
I do agree it makes sense fornscale, but if your data fits in memory Flyte native constructs shine. For example it will ensure your data is stored / serialized correctly. Allows you to use polars, vaex, duckdb etc. tbh I am a huge prop
5.
by kumare3 4y ago
Checkout Flyte.org and it’s sibling project https://www.union.ai/unionml
6.
by kumare3 4y ago
@tomrod, thank you for the callout. By the way we are integrating mlflow into Flyte in a way that you do not need to start the web server to view the logs. They are available Locally and statically in Flyte Ui. Ofcourse you cal also use mlf
7.
by kumare3 4y ago
Some of these were the core problems that we wanted to address as part of https://flyte.org . We started with a team first and multi-tenant approach at the core. For example, each team can have separate IAM roles, secrets are res
8.
by kumare3 4y ago
I remember having this feeling a few years ago. What I realized is that airflow has taught us a few bad habits and also brought ahead an interesting paradigm of the vertical workflow engine. I agree airflow is old, legacy and ideally folks
9.
by kumare3 4y ago
This is cool, you could provide resilience to Red Engine, by providing it a backend using Flyte.org. Checkout example of making a new API on top of Flyte which has similarish feel - https://unionml.readthedocs.io/en/lat
10.
by kumare3 4y ago
+1
11.
by kumare3 4y ago
Thank you. So at Lyft, there exists an AirflowFlyteOperator. It was designed to interop with Airflow. I know the community is working on open sourcing it. Keep a lookout for the same
12.
Orchestrating Data Pipelines at Lyft: Comparing Flyte and Airflow(eng.lyft.com)
15 points by kumare3 4y ago|2 comments
13.
Lyft: Flyte Joins LF AI and Data(eng.lyft.com)
15 points by kumare3 6y ago|0 comments
14.
by kumare3 6y ago
Ohh you are welcome, join the slack channel and ask for help. The community is growing everyday - here are some examples of using it in python https://flytecookbook.readthedocs.io/en/latest/
15.
by kumare3 6y ago
Have you tried Flyte.org?
16.
by kumare3 6y ago
Spotify is moving to https://Flyte.org and building a Luigi to Flyte compiler. Stay tuned
17.
by kumare3 6y ago
Have you tried Flyte - flyte.org?
18.
by kumare3 7y ago
Roberto, this is absolutely one of our goals. When we started, it was with python2.7 still around. We would love contributions, ofcourse we will work with you and adapt it
19.
by kumare3 7y ago
Another great question. So Airflow is used at Lyft for ETL. I think for traditional ETL it still is a good fit. But, there is an effort to not just migrate, but rethink how we can leverage Flyte's capabilities to improve our ETL experi
20.
by kumare3 7y ago
Great question, I am working on a follow up blog that will explain the differences in more detail. Flyte does take some inspiration from airflow, but it has a lot of important differences - Flyte natively understands data flow between tas