5 ms·

I would add these gotchas/recommendations: - Airflow the ETL framework is quite bad. Just use Airflow the scheduler/orchestrator: delegate the actual data tran

by simo7 6y ago

I would add these gotchas/recommendations: - Airflow the ETL framework is quite bad. Just use Airflow the scheduler/orchestrator: delegate the actual data transformation to external services (serverless, kubernetes etc.). - Don't use it for tasks that don't require idempotency (eg. a job that uses a bookmark). - Don't use it for latency-sensitive jobs (this one should be obvious). - Don't use sensors or cross-DAG dependencies. So yeah unfortunately it's not a good fit for all the use cases, but it has the right set of features for some of the most common batch workloads. Also python as the DAG configuration language was a very successful idea, maybe the most important contributor to Airflow success.

bosie 6y ago
What do you mean by bookmark in this context?
- vorpalhex 6y ago
  "I have processed 5/15 records, next run I need to start at record 6". Bookmarking is a common concept for working through a large job in several small runs.
  - devonkim 6y ago
    I wonder if this difference in jargon has origins in different sub-fields. I recognize that concept as "checkpoints" across different companies but I also remember seeing the term from data science folks and thinking that I just don't know the concept.
    - nickpeterson 6y ago
      Often called a Waterline/Watermark as well.
      - sk5t 6y ago
        I'd call it a watermark too; "checkpoint" can mean an intermediate point in a multi-step database transaction.
      - jstarfish 6y ago
        I've always called them offsets.
        wiml 6y ago
        I would call it "a read cursor (that's updated within the transaction)", but I'm boring.
        unixhero 6y ago
        Cursor nomenclature is used for describing where in a sequence the operator is during a code execution pass in database query or similar right? I think I kind of immediately understand "cursor" the best in this context. I also agree, it's a little boring and definitely old school:).
IanCal 6y ago
This is handy thanks - I deal mostly with luigi and this helps me place airflow a bit better.
hn2017 6y ago
If using a docker or kubernetes operator, you can't make use of operators or Airflow connections. Do you just work without those?
- simo7 6y ago
  Correct, I use operators only to delegate the actual workload to an external service.
  - hn2017 6y ago
    Got it, to be clear - do you still use the Airflow for storing connections, or no? If no, how do you store your credentials? We've only done a POC and we've discovered a higher than expected learning curve.
prions 6y ago
> - Don't use it for tasks that don't require idempotency (eg. a job that uses a bookmark). You can totally design your tasks to be idempotent - but its up to you to make them that way. The scheduler or executor doesn't have any context into your job. This is why I encourage people to use a unified base operator and then pass their own docker containers to it. Aka like how https://medium.com/bluecore-engineering/were-all-using-airflow-wrong-and-how-to-fix-it-a56f14cb0753 https://medium.com/bluecore-engineering/were-all-using-airfl... outlines it. > - Don't use it for latency-sensitive jobs (this one should be obvious). IIRC this is being addressed in Airflow 2.0 > - Don't use sensors or cross-DAG dependencies. This is a little extreme. I've never ran into issues with cross dag dependencies or sensors. They make managing my DAGs way easier because wee can separate computation dags from loading dags. context: I built/manage my company's Airflow platform. Everything is managed on k8s.
- simo7 6y ago
  > You can totally design your tasks to be idempotent Yes of course, I mean Airflow is not a good fit for the tasks you don't want to be idempotent (I think most but not all tasks should be idempotent). > I've never ran into issues with cross dag dependencies I believe Airflow docs advice against them when possible. I see why from my experience: less visibility and more complexity, especially for backfills.
- domenp 6y ago
  > context: I built/manage my company's Airflow platform. Everything is managed on k8s My team is running Airflow on a single node but we're slowly outgrowing this setup. We're considering running jobs on k8s. Curious what's your setup like? Is your cluster of a fixed size or does it scale with the load?
- ForHackernews 6y ago
  Using the KubernetesPodOperator for everything adds a huge amount of overhead. You still need Airflow worker nodes, but they're just babysitting the K8S pods doing the real work. I know it's 2020 and memory is cheap or whatever, but Airflow is shockingly wasteful of system resources.
contravariant 6y ago
Sensors are quite helpful when you don't know the exact moment your data will come in.
- simo7 6y ago
  I think it's better to place everything in one DAG if that solves the problem. If it doesn't then sensors are ok I guess, but I would try to avoid them otherwise.
  - prions 6y ago
    I really disagree here. Monolithic dags are a bigger pain to manage. Breaking them out into smaller dags makes retrying/backfilling/etc a lot more straightforward. It also lets you reuse those pieces more easily. We have compute DAGs that are the upstream dependency for many other dags. Originally, this dag was monolithic and loaded data into one table. But because the dag is split into computation and loading we can easily add more downstream dags without changing how the first one operated.
rywalker 6y ago
re: ETL framework - you can get a lot done with the built-in Airflow Operators, including the PythonOperator (and bring in any python dependency you like) and BashOperator (call any CLI, etc.) - it's not drag-and-drop, but I've found it to be quite versitile. re: idempotency - yes, make your workflow tasks idempotent. re: latency - this is being worked on very actively. Ash (PMC member) has committed to working on task latency almost exclusively until it's resolved re: sensors, there is some great work from Airbnb to improve: https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-17+Airflow+sensor+optimization https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-17+A...

by simo7 6y ago

bosie 6y ago

What do you mean by bookmark in this context?

vorpalhex 6y ago
"I have processed 5/15 records, next run I need to start at record 6". Bookmarking is a common concept for working through a large job in several small runs.
- devonkim 6y ago
  I wonder if this difference in jargon has origins in different sub-fields. I recognize that concept as "checkpoints" across different companies but I also remember seeing the term from data science folks and thinking that I just don't know the concept.
  - nickpeterson 6y ago
    Often called a Waterline/Watermark as well.
    - sk5t 6y ago
      I'd call it a watermark too; "checkpoint" can mean an intermediate point in a multi-step database transaction.
    - jstarfish 6y ago
      I've always called them offsets.
      - wiml 6y ago
        I would call it "a read cursor (that's updated within the transaction)", but I'm boring.
        unixhero 6y ago
        Cursor nomenclature is used for describing where in a sequence the operator is during a code execution pass in database query or similar right? I think I kind of immediately understand "cursor" the best in this context. I also agree, it's a little boring and definitely old school:).

IanCal 6y ago

This is handy thanks - I deal mostly with luigi and this helps me place airflow a bit better.

hn2017 6y ago

If using a docker or kubernetes operator, you can't make use of operators or Airflow connections. Do you just work without those?

simo7 6y ago
Correct, I use operators only to delegate the actual workload to an external service.
- hn2017 6y ago
  Got it, to be clear - do you still use the Airflow for storing connections, or no? If no, how do you store your credentials? We've only done a POC and we've discovered a higher than expected learning curve.

prions 6y ago

> - Don't use it for tasks that don't require idempotency (eg. a job that uses a bookmark). You can totally design your tasks to be idempotent - but its up to you to make them that way. The scheduler or executor doesn't have any context into your job. This is why I encourage people to use a unified base operator and then pass their own docker containers to it. Aka like how https://medium.com/bluecore-engineering/were-all-using-airflow-wrong-and-how-to-fix-it-a56f14cb0753 https://medium.com/bluecore-engineering/were-all-using-airfl... outlines it. > - Don't use it for latency-sensitive jobs (this one should be obvious). IIRC this is being addressed in Airflow 2.0 > - Don't use sensors or cross-DAG dependencies. This is a little extreme. I've never ran into issues with cross dag dependencies or sensors. They make managing my DAGs way easier because wee can separate computation dags from loading dags. context: I built/manage my company's Airflow platform. Everything is managed on k8s.

simo7 6y ago
> You can totally design your tasks to be idempotent Yes of course, I mean Airflow is not a good fit for the tasks you don't want to be idempotent (I think most but not all tasks should be idempotent). > I've never ran into issues with cross dag dependencies I believe Airflow docs advice against them when possible. I see why from my experience: less visibility and more complexity, especially for backfills.
domenp 6y ago
> context: I built/manage my company's Airflow platform. Everything is managed on k8s My team is running Airflow on a single node but we're slowly outgrowing this setup. We're considering running jobs on k8s. Curious what's your setup like? Is your cluster of a fixed size or does it scale with the load?
ForHackernews 6y ago
Using the KubernetesPodOperator for everything adds a huge amount of overhead. You still need Airflow worker nodes, but they're just babysitting the K8S pods doing the real work. I know it's 2020 and memory is cheap or whatever, but Airflow is shockingly wasteful of system resources.

contravariant 6y ago

Sensors are quite helpful when you don't know the exact moment your data will come in.

simo7 6y ago
I think it's better to place everything in one DAG if that solves the problem. If it doesn't then sensors are ok I guess, but I would try to avoid them otherwise.
- prions 6y ago
  I really disagree here. Monolithic dags are a bigger pain to manage. Breaking them out into smaller dags makes retrying/backfilling/etc a lot more straightforward. It also lets you reuse those pieces more easily. We have compute DAGs that are the upstream dependency for many other dags. Originally, this dag was monolithic and loaded data into one table. But because the dag is split into computation and loading we can easily add more downstream dags without changing how the first one operated.

rywalker 6y ago

re: ETL framework - you can get a lot done with the built-in Airflow Operators, including the PythonOperator (and bring in any python dependency you like) and BashOperator (call any CLI, etc.) - it's not drag-and-drop, but I've found it to be quite versitile. re: idempotency - yes, make your workflow tasks idempotent. re: latency - this is being worked on very actively. Ash (PMC member) has committed to working on task latency almost exclusively until it's resolved re: sensors, there is some great work from Airbnb to improve: https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-17+Airflow+sensor+optimization https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-17+A...