Turning complex datasets into meaningful analysis requires robust data infrastructure and seamless orchestration. In this episode, we’re joined by Jennifer Melot, Technical Lead at the Center for Security and Emerging Technology (CSET) at Georgetown University, to explore how Airflow powers data-driven insights in technology policy research. Jennifer shares how her team automates workflows to support analysts in navigating complex datasets. 

Key Takeaways:



(02:04) CSET provides data-driven analysis to inform government decision-makers.

(03:54) ETL pipelines merge multiple data sources for more comprehensive insights.

(04:20) Airflow is central to automating and streamlining large-scale data ingestion.

(05:11) Larger-scale databases create challenges that require scalable solutions.

(07:20) Dynamic DAG generation simplifies Airflow adoption for non-engineers.

(12:13) DAG Factory and dynamic task mapping can improve workflow efficiency.

(15:46) Tracking data lineage helps teams understand dependencies across DAGs.

(16:14) New Airflow features enhance visibility and debugging for complex pipelines.

Resources Mentioned:

Jennifer Melot -

https://www.linkedin.com/in/jennifer-melot-aa710144/

Center for Security and Emerging Technology (CSET) -

https://www.linkedin.com/company/georgetown-cset/

Apache Airflow -

https://airflow.apache.org/

Zenodo -

https://zenodo.org/

OpenLineage -

https://openlineage.io/

Cloud Dataplex -

https://cloud.google.com/dataplex

Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Automation #Airflow #MachineLearning

Data orchestration is evolving rapidly, with dynamic workflows becoming the cornerstone of modern data engineering. In this episode, we are joined by Samyak Jain, Senior Software Engineer - Big Data at 99acres.com. Samyak shares insights from his journey with Apache Airflow, exploring how his team built a self-service platform that enables non-technical teams to launch data pipelines and marketing campaigns seamlessly.

Key Takeaways:

(02:02) Starting a career in data engineering by troubleshooting Airflow pipelines.

(04:27) Building self-service portals with Airflow as the backend engine.

(05:34) Utilizing API endpoints to trigger dynamic DAGs with parameterized templates.

(09:31) Managing a dynamic environment with over 1,400 active DAGs.

(11:14) Implementing fault tolerance by segmenting data workflows into distinct layers.

(14:15) Tracking and optimizing query costs in AWS Athena to save $7K monthly.

(16:22) Automating cost monitoring with real-time alerts for high-cost queries.

(17:15) Streamlining Airflow metadata cleanup to prevent performance bottlenecks.

(21:30) Efficiently handling one-time and recurring marketing campaigns using Airflow.

(24:18) Advocating for Airflow features that improve resource management and ownership tracking.

Resources Mentioned:

Samyak Jain -

https://www.linkedin.com/in/samyak-jain-ab5830169/

99acres.com -

https://www.linkedin.com/company/99acres/

Apache Airflow -

https://airflow.apache.org/

AWS Athena -

https://aws.amazon.com/athena/

Kafka -

https://kafka.apache.org/

Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Automation #Airflow #MachineLearning

Testing autonomous vehicles demands precision, scalability and powerful orchestration tools — enter Apache Airflow, a key component of Bosch’s cutting-edge testing framework. In this episode, we sit down with Jens Scheffler, Test Execution Cluster Technical Architect, and Christian Schilling, Product Owner Open Loop Testing Automated Driving, both at Bosch, to explore how Bosch harnesses Airflow to streamline complex testing scenarios. They share insights on scaling workflows, integrating hybrid infrastructures and ensuring vehicle safety through rigorous automated testing.

Key Takeaways:

(01:35) Airflow orchestrates millions of test hours for autonomous systems.

(03:15) Jens scales distributed systems with Kubernetes for job orchestration.

(06:02) Airflow runs hundreds of tests simultaneously.

(06:44) Virtual testing reduces costs and on-road trials.

(12:19) Unified APIs and GUIs streamline operations.

(15:05) Self-service setups empower Bosch teams.

(18:00) Physical hardware integration ensures real-world timing.

(20:30) Dynamic task mapping scales workflows efficiently.

(25:22) Open-source contributions improve stability.

(31:06) Edge and Celery executors power Bosch's hybrid scheduling.

Resources Mentioned:

Jens Scheffler -

https://www.linkedin.com/in/jens-scheffler/

Christian Schilling -

https://www.linkedin.com/in/christian-schilling-a5078831a/

Bosch -

https://www.linkedin.com/company/bosch/

Apache Airflow -

https://airflow.apache.org/

Kubernetes -

https://kubernetes.io

GitHub -

https://github.com

Edge Executor -

https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/executor/index.html

Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Automation #Airflow #MachineLearning

Scaling a data orchestration platform to manage thousands of tasks daily demands innovative solutions and strategic problem-solving. In this episode, we explore the complexities of scaling Airflow and the challenges of orchestrating thousands of tasks in dynamic data environments. Jonathan Rainer, Former Platform Engineer at Monzo Bank, joins us to share his journey optimizing data pipelines, overcoming UI limitations and ensuring DAG consistency in high-stakes scenarios. 

Key Takeaways:

(03:11) Using Airflow to schedule computation in BigQuery.

(07:02) How DAGs with 8,000+ tasks were managed nightly.

(08:18) Ensuring accuracy in regulatory reporting for banking.

(11:35) Handling task inconsistency and DAG failures with automation.

(16:09) Building a service to resolve DAG consistency issues in Airflow.

(25:05) Challenges with scaling the Airflow UI for thousands of tasks.

(27:03) The role of upstream and downstream task management in Airflow.

(37:33) The importance of operational metrics for monitoring Airflow health.

(39:19) Balancing new tools with root cause analysis to address scaling issues.

(41:35) Why scaling solutions require both technical and leadership buy-in

Resources Mentioned:

Jonathan Rainer -

https://www.linkedin.com/in/jonathan-rainer/

Monzo Bank -

https://www.linkedin.com/company/monzo-bank/

Apache Airflow -

https://airflow.apache.org/

BigQuery -

https://airflow.apache.org/docs/apache-airflow-providers-google/stable/operators/cloud/bigquery.html

Kubernetes -

https://kubernetes.io/

Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Automation #Airflow #MachineLearning

The future of data engineering lies in seamless orchestration and automation. In this episode, Arjun Anandkumar, Data Engineer at Telia, shares how his team uses Airflow to drive analytics and AI workflows. He highlights the challenges of scaling data platforms and how adopting best practices can simplify complex processes for teams across the organization. Arjun also discusses the transformative role of tools like Cosmos and Terraform in enhancing efficiency and collaboration. 

Key Takeaways:

(02:16) Telia operates across the Nordics and Baltics, focusing on telecom and energy services.

(03:45) Airflow runs dbt models seamlessly with Cosmos on AWS MWAA.

(05:47) Cosmos improves visibility and orchestration in Airflow.

(07:00) Medallion Architecture organizes data into bronze, silver and gold layers.

(08:34) Task group challenges highlight the need for adaptable workflows.

(15:04) Scaling managed services requires trial, error and tailored tweaks.

(19:46) Terraform scales infrastructure, while YAML templates manage DAGs efficiently.

(20:00) Templated DAGs and robust testing enhance platform management.

(24:15) Open-source resources drive innovation in Airflow practices.

Resources Mentioned:

Arjun Anandkumar -

https://www.linkedin.com/in/arjunanand1/?originalSubdomain=dk

Telia -

https://www.linkedin.com/company/teliacompany/

Apache Airflow -

https://airflow.apache.org/

Cosmos by Astronomer -

https://www.astronomer.io/cosmos/

Terraform -

https://www.terraform.io/

Medallion Architecture by Databricks -

https://www.databricks.com/glossary/medallion-architecture

Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Automation #Airflow #MachineLearning

Transforming bottlenecked finance processes into streamlined, automated systems requires the right tools and a forward-thinking approach. In this episode, Mihir Samant, Senior Data Analyst at Etraveli Group, joins us to share how his team leverages Airflow to revolutionize finance automation. With extensive experience in data workflows and a passion for open-source tools, Mihir provides valuable insights into building efficient, scalable systems. We explore the transformative power of Airflow in automating workflows and enhancing data orchestration within the finance domain. 

Key Takeaways:

(02:14) Etraveli Group specializes in selling affordable flight tickets and ancillary services.

(03:56) Mihir’s finance automation team uses Airflow to tackle month-end bottlenecks.

(06:00) Airflow's flexibility enables end-to-end automation for finance workflows.

(07:00) Open-source Airflow tools offer cost-effective solutions for new teams.

(08:46) Sensors and dynamic DAGs are pivotal features for optimizing tasks.

(13:30) GitSync simplifies development by syncing environments seamlessly.

(16:27) Plans include integrating Databricks for more advanced data handling.

(17:58) Airflow and Databricks offer multiple flexible methods to trigger workflows and execute SQL queries seamlessly.

Resources Mentioned:

Mihir Samant -

https://www.linkedin.com/in/misamant/?originalSubdomain=ca

Etraveli Group -

https://www.linkedin.com/company/etraveli-group/

Apache Airflow -

https://airflow.apache.org/

Docker -

https://www.docker.com/

Databricks -

https://www.databricks.com/

Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Automation #Airflow #MachineLearning

Data engineering is entering a new era, where orchestration and automation are redefining how large-scale projects operate. This episode features Vasantha Kosuri-Marshall, Data and ML Ops Engineer at Ford Motor Company. Vasantha shares her expertise in managing complex data pipelines. She takes us through Ford's transition to cloud platforms, the adoption of Airflow and the intricate challenges of orchestrating data in a diverse environment.

Key Takeaways:

(03:10) Vasantha’s transition to the Advanced Driving Assist Systems team at Ford.

(05:42) Early adoption of Airflow to orchestrate complex data pipelines.

(09:29) Ford's move from on-premise data solutions to Google Cloud Platform.

(12:03) The importance of Airflow's scheduling capabilities for efficient data management.

(16:12) Using Kubernetes to scale Airflow for large-scale data processing.

(19:59) Vasantha’s experience in overcoming challenges with legacy orchestration tools.

(22:22) Integration of data engineering and data science pipelines at Ford.

(28:03) How deferrable operators in Airflow improve performance and save costs.

(32:12) Vasantha’s insights into tuning Airflow properties for thousands of DAGs.

(36:09) The significance of monitoring and observability in managing Airflow instances.

Resources Mentioned:

Vasantha Kosuri-Marshall -

https://www.linkedin.com/in/vasantha-kosuri-marshall-0b0aab188/

Apache Airflow -

https://airflow.apache.org/

Google Cloud Platform (GCP) -

https://cloud.google.com/

Ford Motor Company | LinkedIn -

https://www.linkedin.com/company/ford-motor-company/

Ford Motor Company | Website -

https://www.ford.com/

Astronomer -

https://www.astronomer.io/

Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Automation #Airflow #MachineLearning

Data is the backbone of every modern business, but unlocking its full potential requires the right tools and strategies. In this episode, Ryan Delgado, Director of Engineering at Ramp, joins us to explore how innovative data platforms can transform business operations and fuel growth. He shares insights on integrating Apache Airflow, optimizing data workflows and leveraging analytics to enhance customer experiences.

Key Takeaways:



(01:52) Data is the lifeblood of Ramp, touching every vertical in the company.

(03:18) Ramp’s data platform team enables high-velocity scaling through tailored tools.

(05:27) Airflow powers Ramp’s enterprise data warehouse integrations for advanced analytics.

(07:55) Centralizing data in Snowflake simplifies storage and analytics pipelines.

(12:08) Machine learning models at Ramp integrate seamlessly with Airflow for operational excellence.

(14:11) Leveraging Airflow datasets eliminates inefficiencies in DAG dependencies.

(17:22) Platforms evolve from solving narrow business problems to scaling organizationally.

(18:55) ClickHouse enhances Ramp’s OLAP capabilities with 100x performance improvements.

(19:47) Ramp’s OLAP platform improves performance by reducing joins and leveraging ClickHouse.

(21:46) Ryan envisions a lighter-weight, more Python-native future for Airflow.

Resources Mentioned:

Ryan Delgado -

https://www.linkedin.com/in/ryan-delgado-69544568/

Ramp -

https://www.linkedin.com/company/ramp/

Apache Airflow -

https://airflow.apache.org/

Snowflake -

https://www.snowflake.com/

ClickHouse -

https://clickhouse.com/

dbt -

https://www.getdbt.com/

Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Automation #Airflow #MachineLearning

What does it take to go from fixing a broken link to becoming a committer for one of the world’s leading open-source projects? 

Amogh Desai, Senior Software Engineer at Astronomer, takes us through his journey with Apache Airflow. From small contributions to building meaningful connections in the open-source community, Amogh’s story provides actionable insights for anyone on the cusp of their open-source journey.

Key Takeaways:

(02:09) Building data engineering platforms at Cloudera with Kubernetes.

(04:00) Brainstorming led to contributing to Apache Airflow.

(05:17) Starting small with link fixes, progressing to Breeze development.

(07:00) Becoming a committer for Apache Airflow in September 2023.

(09:51) The steep learning curve for contributing to Airflow.

(16:30) Using GitHub’s “good-first-issue” label to get started.

(18:15) Setting up a development environment with Breeze.

(22:00) Open-source contributions enhance your resume and career.

(24:51) Amogh’s advice: Start small and stay consistent.

(28:12) Engage with the community via Slack, email lists and meetups.

Resources Mentioned:

Amogh Desai -

https://www.linkedin.com/in/amogh-desai-385141157/?originalSubdomain=in%20%20https://www.linkedin.com/company/astronomer/

Astronomer -

https://www.linkedin.com/company/astronomer/

Apache Airflow GitHub Repository -

https://github.com/apache/airflow

Contributors Quick Guide -

https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst

Breeze Development Tool -

https://github.com/apache/airflow/tree/main/dev/breeze

Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Automation #Airflow #MachineLearning

Data orchestration and machine learning are shaping how organizations handle massive datasets and drive customer-focused strategies. Tools like Apache Airflow are central to this transformation. In this episode, Vasyl Vasyuta, R&D Team Leader at Optimove, joins us to discuss how his team leverages Airflow to optimize data processing, orchestrate machine learning models and create personalized customer experiences.

Key Takeaways:



(01:59) Optimove tailors marketing notifications with personalized customer journeys.

(04:25) Airflow orchestrates Snowflake procedures for massive datasets.

(05:11) DAGs manage workflows with branching and replay plugins.

(05:41) The "Joystick" plugin enables seamless data replays.

(09:33) Airflow supports MLOps for customer data grouping.

(11:15) Machine learning predicts customer behavior for better campaigns.

(13:20) Thousands of DAGs run every five minutes for data processing.

(15:36) Custom versioning allows rollbacks and gradual rollouts.

(18:00) Airflow logs enhance operational observability.

(23:00) DAG versioning in Airflow 3.0 could boost efficiency.

Resources Mentioned:

Vasyl Vasyuta -

https://www.linkedin.com/in/vasyl-vasyuta-3270b54a/

Optimove -

https://www.linkedin.com/company/optimove/

Apache Airflow -

https://airflow.apache.org/

Snowflake -

https://www.snowflake.com/

Datadog -

https://www.datadoghq.com/

Apache Airflow Survey -

https://astronomer.typeform.com/airflowsurvey24

Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Automation #Airflow #MachineLearning

Bridging the gap between data teams and business priorities is essential for maximizing impact and building value-driven workflows. Katie Bauer, Senior Director of Data at GlossGenius, joins us to share her principles for creating effective, aligned data teams. In this episode, Katie draws from her experience at GlossGenius, Reddit and Twitter to highlight the common pitfalls data teams face and how to overcome them. She offers practical strategies for aligning team efforts with organizational goals and fostering collaboration with stakeholders.

 Key Takeaways:

(02:36) GlossGenius provides an all-in-one platform for beauty professionals.

(03:59) Airflow orchestrates data and MLOps workflows at GlossGenius.

(04:41) Focusing on value helps data teams achieve greater impact.

(06:23) Aligning team priorities with company goals minimizes friction.

(08:44) Building strong stakeholder relationships requires curiosity.

(12:46) Treating roles as flexible fosters team innovation.

(13:21) Adapting to new technologies improves effectiveness.

(18:28) Acting like your time is valuable earns respect.

(23:38) Proactive data initiatives drive strategic value.

(24:20) Usage data offers critical insights into tool effectiveness.

Resources Mentioned:

Katie Bauer -

https://www.linkedin.com/in/mkatiebauer/

GlossGenius -

https://www.linkedin.com/company/glossgenius/

Apache Airflow -

https://airflow.apache.org/

DBT -

https://www.getdbt.com/

Cosmos -

https://cosmos.apache.org/

Apache Airflow Survey -

https://astronomer.typeform.com/airflowsurvey24

Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Automation #Airflow #MachineLearning