Airflow is a powerful platform to automate and manage workflows. There are several options to deploy Airflow on AWS, including MWAA, ECS OR EKS:
To quickly launch spark and kafka for local development, docker would be the top choice as of its flexibily and isolated environment. This save lot of time for manually installing bunch of packages as well as conflicting issues.
Along with the widely cloud adoption, integrated development environment (IDE) on browser is a need to boot developers’ productivity. People can collaboratively view, edit and commit on any devices with internet accessed browser. Additionally, you’re no longer worry about setting up your local development config. You can consider Cloud9 (AWS) or paid service like codeanywhere.
Grafana is an open-source web application specialized in time-series visualization. Hence, it is suitable for the the purpose of monitoring. One of interesting facts is that Grafana is a fork from Kibana 3 to enhance the ability of dashboard editing and make it as a clean and elegant time-series visualization tool. To get the sense of Grafana dashboard, surf this link: https://play.grafana.org.
Update your browser to view this website correctly. Update my browser now