6 ms·

Apache Airflow seems like a really interesting project but I don't know anyone using that can give a real life pros/cons to it. Anyone here dares to give some

Apache Airflow seems like a really interesting project but I don't know anyone using that can give a real life pros/cons to it. Anyone here dares to give some feedback in that sense? Ps: Why do people still use Prezi? It gives me vertigo.

Maro 8y ago
We base our whole DS infrastructure on Airflow (and Superset): http://bytepawn.com/fetchr-airflow.html http://bytepawn.com/fetchr-airflow.html http://bytepawn.com/fetchr-data-science-infra.html http://bytepawn.com/fetchr-data-science-infra.html Airflow is somewhere between good enough and pretty cool, it's based on what we had at Facebook (called Dataswarm). IMO in 2-3 years Airflow will be the de-facto ETL standard, like Hadoop used to be for "Big data". If you're rolling your own ETL at this point, you're wasting your time. If you're using something else, you're (probably) missing out on ETL-as-code goodness.
- rhombocombus 8y ago
  How does it compare to big iron enterprise ETL tools like IBM datastage? I have only dabbled but it looks far more appealing for a variety of reasons.
  - philosopherlawr 8y ago
    ETL tools just can't compete with a tool that forces code to do anything. It might seem backwards, but we've abandoned all non-code environments and force pure-code for configuration for all of our pipelines.
  - caravel 8y ago
    For context, I used Datastage, Informatica, Ab Initio and SSIS in previous lives an went on to write the first version of Airflow. I developed a taste for pipelines-as-code while working at Facebook using an internal tool that is not open source. I'd argue that pipelines as code, as opposed to dragndrop GUIs, is a better approach, at least for people who are comfortable writing code. Code is easy to version, test, diff, collaborate and allows for the creation of arbitrary abstractions.
- sweml 8y ago
  IMO Airflow currently is the de-facto standard and in 2-3 years it will go the way of Hadoop
- thisisit 8y ago
  > like Hadoop used to be for "Big data" So, you are saying it will be going with time, after it becomes the de-facto ETL tool?
politelemon 8y ago
We consider it a glorified cron replacement. The main selling point is its scheduling feature and the ability to view logs via the web UI it provides. You write DAGs in Python to do 'stuff', schedule it to run, say, every hour. You can then get a history of its runs, failures, what went wrong. Rerun things if needed. Those are the pros. Cons - when new devs try to treat it as a programming paradigm, things can get difficult to work with. Some aspects aren't easily automatable - eg creating users. Needs to make its authentication options obvious and would be good to have some finer grain control over who can do what in the Airflow UI. Overall we're quite happy with it and also using it for datascience as well as data feeds, data workflows, ETL processes.
- yoquan 8y ago
  There is an attempt to create a role based access control by the guys at WePay according to the shared slide "RBAC talk" below [1]. Don't know why their repo [2] can't be accessed now, though. [1] https://www.meetup.com/ja-JP/Bay-Area-Apache-Airflow-Incubating-Meetup/events/244525050/ https://www.meetup.com/ja-JP/Bay-Area-Apache-Airflow-Incubat... [2] https://github.com/wepay/airflow-webserver https://github.com/wepay/airflow-webserver
  - tedmiston 8y ago
    The RBAC UI has since been merged into Airflow and is now released in Airflow 1.10. https://github.com/apache/incubator-airflow/tree/master/airflow/www_rbac https://github.com/apache/incubator-airflow/tree/master/airf... https://github.com/apache/incubator-airflow/commit/05e1861e24de42f9a2c649cd93041c5c744504e1#diff-5aeaca39c3e310ff9883f8ac4d506d1d https://github.com/apache/incubator-airflow/commit/05e1861e2... To enable it, set `rbac = True` under the `webserver` group in your airflow.cfg, or via env var: export AIRFLOW__WEBSERVER__RBAC=True
    - yoquan 8y ago
      That's great to know, thank you. Eager to try 1.10!
boulos 8y ago
Disclosure: I work on Google Cloud. We ended up choosing Airflow for our managed workflow service, Composer [1]. One of the main cons is that this space is generally fragmented. When you’re starting out with a simple repeatable task, the natural default is just a cron job (and if you’re “fancy”, using Jenkins or similar to kick it off). By comparison, getting started with Airflow means setting it all up, and then writing a Python script that represents your DAG. It may be better hygiene, but that’s not people’s first preference. Beyond the obvious “this thing is a real workflow orchestration system, it handles failures and dependencies”, I think the main advantage is the pre-built Operators. Instead of being just given a blank bash or python script, this is a community-driven effort to avoid everyone needing to roll their own. It’s still a young community, but growing quickly (and we’re intent on pushing). [1] https://cloud.google.com/composer/ https://cloud.google.com/composer/
- tedmiston 8y ago
  Disclosure: I work on Astronomer.io, an open source Airflow platform and SaaS [1][2], and also contribute to Airflow. Our experience was similar to boulous' — Airflow is awesome once it's running but getting it running in an environment that scales to the point that you can deploy your first production DAG can take some effort. That's what led us to trying to do that work once for everyone. Reusability and composability of components are some of my favorite aspects of working with Airflow. [1]: https://www.astronomer.io/ https://www.astronomer.io/ [2]: https://github.com/astronomerio/helm.astronomer.io https://github.com/astronomerio/helm.astronomer.io
glogla 8y ago
We tested it, but the performancd was bad. We needed hundred workflows with few hundred taks each, and Airflow would just topple over daily. We ended up with proprietary tool from Teradata thats basically Airflow written in perl - but it can handle all the work. Other than scalability, Airflow is pretty nice.
- tpaschalis 8y ago
  It sounds weird that you had these problems. Silly question, but might it have been a database optimization issue?
- vgy7ujm 8y ago
  Perl saved the day again.
- caravel 8y ago
  [full disclosure, I'm the creator of Airflow] Many environments run tens of thousands of concurrent tasks, and hundreds of thousands of tasks daily. The list of companies using Airflow speaks for itself https://github.com/apache/incubator-superset#who-uses-apache-superset-incubating https://github.com/apache/incubator-superset#who-uses-apache... But hey, it's like anything, you have to do a bit of work to get distributed systems to run at scale. There are now hosted solutions to help with that (Google Cloud Composer and Astronomer.io)
  - tedmiston 8y ago
    https://github.com/apache/incubator-airflow#who-uses-airflow https://github.com/apache/incubator-airflow#who-uses-airflow :)
- tedmiston 8y ago
  I run an Airflow instance that does millions of tasks per month across dozens of DAGs. There's some performance tuning involved in the configuration file and of course you need the underlying resources available but Airflow has scaled to this level well for us. If you are able to reproduce and can post to the dev mailing list, we are happy to help... especially so if it gets you off of a proprietary tool written in Perl ;). 1.10 was just released and adds a ton of commits. I'd really encourage you to give Airflow another shot if you have the time.
tpaschalis 8y ago
There's a bunch of different tools to do the same job, from manual cron jobs, to Luigi, Pinball, Azkaban, Oozie, Taverna, Mistral. I've started to use it for personal projects, and slowly probing for adoption in our shop, where applicable. The good points I have seen - It's simple Python, and not XML like Azkaban. I've seen people with less technical expertise build useful stuff quickly, and automate their workflows. - Very good UI, which just lets you do what you need without fuss. - Easy to build modular and interactive flows, with interesting stuff as sensors, communications between operators, triggers etc. - Everything is stored into a database, which I can query about anything related to the processes run and Airflow itself - Its source is grok-able and documented, it allows you to easily add your own modules (or "operators" as they're called) - Many add-on modules for operators already exist from the community - Easier to force the team to version control your process flows Some cons, from the light use I've seen - If you scale beyond a point, you have to take care of scaling the database as well, adding DBA work - I've encountered some issues with scheduler and backfilled jobs, and `depends_on_past`, but it might be my limited experience - People may start to use specific external dependencies/modules, which you will then need to keep track of - Uses its own lingo/terminology, which you'll have to learn and use - Uses system time, so no running in different timezones I have high hopes for the project, as it's currently incubating for the Apache Foundation, and I hope it remains minimal and keeps the present scope. If it seems interesting to you, my suggestion is to start small, keep in mind that it handles relations between tasks and not data, and try to automate some easy bash script that you currently handle with cron.
- neuromantik8086 8y ago
  > There's a bunch of different tools to do the same job Yup. Almost, too many tools, in fact. https://s.apache.org/existing-workflow-systems https://s.apache.org/existing-workflow-systems
  - neuromantik8086 8y ago
    As a somewhat related addendum, some of the worst pipelines I've encountered have been in scientific computing. Conceptually, DAGs are quite simple but for some reason things always end up gummed up in implementation, which is partially why scientific results are much harder to reproduce than they should be. The disconnect between how much harder pipeline creation is compared to how easy it should be in the sciences has always confused me a bit.
sejtnjir 8y ago
Adopting Airflow over cron was a huge leap for the analytics company I worked for. I think the biggest win was making the complex net of dependent batch operations explicit. That logic lived inside the head of the lead dev before that point. This allowed management to reason about how new functionality effected operational complexity, which would have been unimaginable before.
monksy 8y ago
If you've ever used a tool similar to HP Operations Orchestration. It's pretty much a stripped down version of that. Cons: - It's not very stable. (It requires a lot of configuration to get it to do more than one process at a time) - It's very easy to get the UI to fall over. - It's very difficult to get tasks+jobs to stop running once they started. (You can delete/stop/cancel a job.. but under the covers it keeps running and your next iteration is going to wait.. if it ever does complete before you can go through the develop, test cycle again) - It's written in Python: Expect to have issues with your environment. The latest version of Airflow doesn't work with 3.7.xish because the async word was made a keyword. There goes that method. - There is no sharing (xcoms is frowned upon) of data from one process to another. This means that if you're trying to pull data from S3, you're going to have to hard code it to a predictable place. The next operation acts completely indepenently and runs that. - The connections between tasks are superficial. They're just there to order it based on how you specified it. Also, it can be a bit difficult to debug when you have multiple layers and multiple depedency declarations where something is both a upstream and downstream of the same depedency. - No optimization. It will not split up the work per task. You have to define that work manually. (See the next complaint) - No Dynamic tasks or Dags. You cannot generate a new dag or task after the dag is initialized. That means that if you have to perform 1 000 000 000 000 API calls, you can't just break that up into 200 api calls per task and then max out your compacity in your workers. - That example that they had of a dag of thousands of tasks. That's a bad practice. Timeouts on dags are going to be reached by the time that completes, and it'll try to restart on a schedule.
- gbrown 8y ago
  Any opinion on NiFi or other open source alternatives? I'm evaluating products in this space, but it seems pretty hard to tell without just giving them a try how well they'd integrate into my work.
  - monksy 8y ago
    At the moment I haven't found any other OSS alternatives. Maybe Chef or ansible. I'll ask my friend Warren.
- 8y ago
framebit 8y ago
On a Data Engineering/Data Science team we were using it as the missing Apache Spark scheduler. At first cron was good enough but then as more projects and processes and people came online we turned to Airflow to help wrangle everything and it helped!
tedmiston 8y ago
Disclosure: I work on Astronomer.io, an open source Airflow platform and SaaS, and also contribute to Airflow. IMO the pros / cons are really relative to what you're comparing it to and what your workflow needs are. We've written some guides on "Airflow vs ___" [1] (currently AWS Glue and Oozie). Feel free to email me (in profile) if you'd like to see a comparison to something else. I like these two posts on Airflow vs Luigi, Pinball, Azkaban, etc [2][3]. [1]: https://www.astronomer.io/guides/ https://www.astronomer.io/guides/ [2]: http://bytepawn.com/luigi-airflow-pinball.html http://bytepawn.com/luigi-airflow-pinball.html [3]: https://robinhood.engineering/why-robinhood-uses-airflow-aed13a9a90c8 https://robinhood.engineering/why-robinhood-uses-airflow-aed...

Maro 8y ago

We base our whole DS infrastructure on Airflow (and Superset): http://bytepawn.com/fetchr-airflow.html http://bytepawn.com/fetchr-airflow.html http://bytepawn.com/fetchr-data-science-infra.html http://bytepawn.com/fetchr-data-science-infra.html Airflow is somewhere between good enough and pretty cool, it's based on what we had at Facebook (called Dataswarm). IMO in 2-3 years Airflow will be the de-facto ETL standard, like Hadoop used to be for "Big data". If you're rolling your own ETL at this point, you're wasting your time. If you're using something else, you're (probably) missing out on ETL-as-code goodness.

rhombocombus 8y ago
How does it compare to big iron enterprise ETL tools like IBM datastage? I have only dabbled but it looks far more appealing for a variety of reasons.
- philosopherlawr 8y ago
  ETL tools just can't compete with a tool that forces code to do anything. It might seem backwards, but we've abandoned all non-code environments and force pure-code for configuration for all of our pipelines.
- caravel 8y ago
  For context, I used Datastage, Informatica, Ab Initio and SSIS in previous lives an went on to write the first version of Airflow. I developed a taste for pipelines-as-code while working at Facebook using an internal tool that is not open source. I'd argue that pipelines as code, as opposed to dragndrop GUIs, is a better approach, at least for people who are comfortable writing code. Code is easy to version, test, diff, collaborate and allows for the creation of arbitrary abstractions.
sweml 8y ago
IMO Airflow currently is the de-facto standard and in 2-3 years it will go the way of Hadoop
thisisit 8y ago
> like Hadoop used to be for "Big data" So, you are saying it will be going with time, after it becomes the de-facto ETL tool?

politelemon 8y ago

We consider it a glorified cron replacement. The main selling point is its scheduling feature and the ability to view logs via the web UI it provides. You write DAGs in Python to do 'stuff', schedule it to run, say, every hour. You can then get a history of its runs, failures, what went wrong. Rerun things if needed. Those are the pros. Cons - when new devs try to treat it as a programming paradigm, things can get difficult to work with. Some aspects aren't easily automatable - eg creating users. Needs to make its authentication options obvious and would be good to have some finer grain control over who can do what in the Airflow UI. Overall we're quite happy with it and also using it for datascience as well as data feeds, data workflows, ETL processes.

yoquan 8y ago
There is an attempt to create a role based access control by the guys at WePay according to the shared slide "RBAC talk" below [1]. Don't know why their repo [2] can't be accessed now, though. [1] https://www.meetup.com/ja-JP/Bay-Area-Apache-Airflow-Incubating-Meetup/events/244525050/ https://www.meetup.com/ja-JP/Bay-Area-Apache-Airflow-Incubat... [2] https://github.com/wepay/airflow-webserver https://github.com/wepay/airflow-webserver
- tedmiston 8y ago
  The RBAC UI has since been merged into Airflow and is now released in Airflow 1.10. https://github.com/apache/incubator-airflow/tree/master/airflow/www_rbac https://github.com/apache/incubator-airflow/tree/master/airf... https://github.com/apache/incubator-airflow/commit/05e1861e24de42f9a2c649cd93041c5c744504e1#diff-5aeaca39c3e310ff9883f8ac4d506d1d https://github.com/apache/incubator-airflow/commit/05e1861e2... To enable it, set `rbac = True` under the `webserver` group in your airflow.cfg, or via env var: export AIRFLOW__WEBSERVER__RBAC=True
  - yoquan 8y ago
    That's great to know, thank you. Eager to try 1.10!

boulos 8y ago

Disclosure: I work on Google Cloud. We ended up choosing Airflow for our managed workflow service, Composer [1]. One of the main cons is that this space is generally fragmented. When you’re starting out with a simple repeatable task, the natural default is just a cron job (and if you’re “fancy”, using Jenkins or similar to kick it off). By comparison, getting started with Airflow means setting it all up, and then writing a Python script that represents your DAG. It may be better hygiene, but that’s not people’s first preference. Beyond the obvious “this thing is a real workflow orchestration system, it handles failures and dependencies”, I think the main advantage is the pre-built Operators. Instead of being just given a blank bash or python script, this is a community-driven effort to avoid everyone needing to roll their own. It’s still a young community, but growing quickly (and we’re intent on pushing). [1] https://cloud.google.com/composer/ https://cloud.google.com/composer/

tedmiston 8y ago
Disclosure: I work on Astronomer.io, an open source Airflow platform and SaaS [1][2], and also contribute to Airflow. Our experience was similar to boulous' — Airflow is awesome once it's running but getting it running in an environment that scales to the point that you can deploy your first production DAG can take some effort. That's what led us to trying to do that work once for everyone. Reusability and composability of components are some of my favorite aspects of working with Airflow. [1]: https://www.astronomer.io/ https://www.astronomer.io/ [2]: https://github.com/astronomerio/helm.astronomer.io https://github.com/astronomerio/helm.astronomer.io

glogla 8y ago

We tested it, but the performancd was bad. We needed hundred workflows with few hundred taks each, and Airflow would just topple over daily. We ended up with proprietary tool from Teradata thats basically Airflow written in perl - but it can handle all the work. Other than scalability, Airflow is pretty nice.

tpaschalis 8y ago
It sounds weird that you had these problems. Silly question, but might it have been a database optimization issue?
vgy7ujm 8y ago
Perl saved the day again.
caravel 8y ago
[full disclosure, I'm the creator of Airflow] Many environments run tens of thousands of concurrent tasks, and hundreds of thousands of tasks daily. The list of companies using Airflow speaks for itself https://github.com/apache/incubator-superset#who-uses-apache-superset-incubating https://github.com/apache/incubator-superset#who-uses-apache... But hey, it's like anything, you have to do a bit of work to get distributed systems to run at scale. There are now hosted solutions to help with that (Google Cloud Composer and Astronomer.io)
- tedmiston 8y ago
  https://github.com/apache/incubator-airflow#who-uses-airflow https://github.com/apache/incubator-airflow#who-uses-airflow :)
tedmiston 8y ago
I run an Airflow instance that does millions of tasks per month across dozens of DAGs. There's some performance tuning involved in the configuration file and of course you need the underlying resources available but Airflow has scaled to this level well for us. If you are able to reproduce and can post to the dev mailing list, we are happy to help... especially so if it gets you off of a proprietary tool written in Perl ;). 1.10 was just released and adds a ton of commits. I'd really encourage you to give Airflow another shot if you have the time.

tpaschalis 8y ago

There's a bunch of different tools to do the same job, from manual cron jobs, to Luigi, Pinball, Azkaban, Oozie, Taverna, Mistral. I've started to use it for personal projects, and slowly probing for adoption in our shop, where applicable. The good points I have seen - It's simple Python, and not XML like Azkaban. I've seen people with less technical expertise build useful stuff quickly, and automate their workflows. - Very good UI, which just lets you do what you need without fuss. - Easy to build modular and interactive flows, with interesting stuff as sensors, communications between operators, triggers etc. - Everything is stored into a database, which I can query about anything related to the processes run and Airflow itself - Its source is grok-able and documented, it allows you to easily add your own modules (or "operators" as they're called) - Many add-on modules for operators already exist from the community - Easier to force the team to version control your process flows Some cons, from the light use I've seen - If you scale beyond a point, you have to take care of scaling the database as well, adding DBA work - I've encountered some issues with scheduler and backfilled jobs, and `depends_on_past`, but it might be my limited experience - People may start to use specific external dependencies/modules, which you will then need to keep track of - Uses its own lingo/terminology, which you'll have to learn and use - Uses system time, so no running in different timezones I have high hopes for the project, as it's currently incubating for the Apache Foundation, and I hope it remains minimal and keeps the present scope. If it seems interesting to you, my suggestion is to start small, keep in mind that it handles relations between tasks and not data, and try to automate some easy bash script that you currently handle with cron.

neuromantik8086 8y ago
> There's a bunch of different tools to do the same job Yup. Almost, too many tools, in fact. https://s.apache.org/existing-workflow-systems https://s.apache.org/existing-workflow-systems
- neuromantik8086 8y ago
  As a somewhat related addendum, some of the worst pipelines I've encountered have been in scientific computing. Conceptually, DAGs are quite simple but for some reason things always end up gummed up in implementation, which is partially why scientific results are much harder to reproduce than they should be. The disconnect between how much harder pipeline creation is compared to how easy it should be in the sciences has always confused me a bit.

sejtnjir 8y ago

Adopting Airflow over cron was a huge leap for the analytics company I worked for. I think the biggest win was making the complex net of dependent batch operations explicit. That logic lived inside the head of the lead dev before that point. This allowed management to reason about how new functionality effected operational complexity, which would have been unimaginable before.

monksy 8y ago

If you've ever used a tool similar to HP Operations Orchestration. It's pretty much a stripped down version of that. Cons: - It's not very stable. (It requires a lot of configuration to get it to do more than one process at a time) - It's very easy to get the UI to fall over. - It's very difficult to get tasks+jobs to stop running once they started. (You can delete/stop/cancel a job.. but under the covers it keeps running and your next iteration is going to wait.. if it ever does complete before you can go through the develop, test cycle again) - It's written in Python: Expect to have issues with your environment. The latest version of Airflow doesn't work with 3.7.xish because the async word was made a keyword. There goes that method. - There is no sharing (xcoms is frowned upon) of data from one process to another. This means that if you're trying to pull data from S3, you're going to have to hard code it to a predictable place. The next operation acts completely indepenently and runs that. - The connections between tasks are superficial. They're just there to order it based on how you specified it. Also, it can be a bit difficult to debug when you have multiple layers and multiple depedency declarations where something is both a upstream and downstream of the same depedency. - No optimization. It will not split up the work per task. You have to define that work manually. (See the next complaint) - No Dynamic tasks or Dags. You cannot generate a new dag or task after the dag is initialized. That means that if you have to perform 1 000 000 000 000 API calls, you can't just break that up into 200 api calls per task and then max out your compacity in your workers. - That example that they had of a dag of thousands of tasks. That's a bad practice. Timeouts on dags are going to be reached by the time that completes, and it'll try to restart on a schedule.

gbrown 8y ago
Any opinion on NiFi or other open source alternatives? I'm evaluating products in this space, but it seems pretty hard to tell without just giving them a try how well they'd integrate into my work.
- monksy 8y ago
  At the moment I haven't found any other OSS alternatives. Maybe Chef or ansible. I'll ask my friend Warren.
8y ago

framebit 8y ago

On a Data Engineering/Data Science team we were using it as the missing Apache Spark scheduler. At first cron was good enough but then as more projects and processes and people came online we turned to Airflow to help wrangle everything and it helped!

tedmiston 8y ago

Disclosure: I work on Astronomer.io, an open source Airflow platform and SaaS, and also contribute to Airflow. IMO the pros / cons are really relative to what you're comparing it to and what your workflow needs are. We've written some guides on "Airflow vs ___" [1] (currently AWS Glue and Oozie). Feel free to email me (in profile) if you'd like to see a comparison to something else. I like these two posts on Airflow vs Luigi, Pinball, Azkaban, etc [2][3]. [1]: https://www.astronomer.io/guides/ https://www.astronomer.io/guides/ [2]: http://bytepawn.com/luigi-airflow-pinball.html http://bytepawn.com/luigi-airflow-pinball.html [3]: https://robinhood.engineering/why-robinhood-uses-airflow-aed13a9a90c8 https://robinhood.engineering/why-robinhood-uses-airflow-aed...