Airflow on EKS

Airflow is a powerful platform to automate and manage workflows. There are several options to deploy Airflow on AWS, including MWAA, ECS OR EKS:

1. Deploying Airflow on AWS Managed Workflow for Apache Airflow (MWAA): This option provides a fully managed service for Apache Airflow and is a good choice for those who want a quick and easy way to get started with Airflow on AWS.

1. Deploying Airflow on AWS Elastic Container Service (ECS): This option allows you to run Airflow in containers, providing a scalable and highly available environment for your workflows. ECS provides a flexible and cost-effective way to manage the deployment and scaling of your applications.
2. Deploying Airflow on AWS Elastic Kubernetes Service (EKS): This option provides a managed Kubernetes environment for deploying, scaling, and operating Airflow. EKS offers the flexibility to run and manage containerized applications, and provides the latest version of Airflow and easier package management.

After considering all the options, we decided to run Airflow on EKS for its flexibility, up-to-date version of Airflow, and easier package management. The diagram below shows how the Airflow Helm chart is deployed on the EKS cluster.

The diagram below depicts Airflow within the context of an EKS cluster.