Large-scale cloud migrations can result in a lot to manage when it comes to orhcestration: multi-cloud targets, on-prem extraction, dynamic workloads, and strict operational controls. In this episode, Kenten Danas is joined by Bhanu Prakash, Technical Lead at HCL Tech, to talk through how his team uses Airflow to orchestrate end-to-end data and platform migrations to the cloud, generate Dags dynamically for micro-batch workloads, and automate incident response through a ServiceNow integration.

Key Takeaways:

00:00 Introduction.

01:19 Bhanu's role as an Airflow and ETL SME at HCL Tech, leading architecture and delivery across data engineering engagements.

02:30 Six years of Airflow use cases: data migration, scheduled analytics, customer segmentation refreshes, and web-application-adjacent data ops.

03:19 When to recommend Airflow to clients: everything except real-time, with cloud-agnostic portability as the deciding factor over ADF and Glue.

06:19 Where Airflow fits in the current cloud migration project: watermarking, cross-reference tables, on-prem to cloud movement, and downstream Databricks and Snowflake processing.

08:01 Running through Airflow versions 2.3, 2.11, and now 3.2, and how backwards compatibility kept 99% of the code unchanged.

11:10 Building a Dag generation utility driven by JSON config, plus runtime Dag creation for micro-batching (illustrated with an e-commerce example).

14:45 Making Dags configurable for clients with pull and push triggering, and the blue-green pattern for promoting Dags through dev, stage, pre-prod, and prod.

16:28 Using Airflow's ServiceNow integration to build an auto-resolution engine that uses LLMs to build context from past incidents and apply gated fixes.

18:12 A wish-list feature: making Airflow an AI-first platform where cross-tool debugging happens inside the Airflow UI.

Resources Mentioned:

• HCL Tech
• Apache Airflow
• ServiceNow
• Databricks
• Snowflake

Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Orchestration #Airflow

At Lyft, a small data engineering team supports dozens of scientists and analysts who tune driver pay configurations on demand, all under SOX controls. In this episode, Ken Obata, Senior Software Engineer at Lyft, joins Kenten Danas to walk through how his team built self-service Dags with an event-driven integration between GitHub Actions and Airflow, and how they solved the data modeling and consistency problems that came with it.

Key Takeaways:

00:00 Introduction.

02:49 How Airflow responsibility is split at Lyft between the infrastructure org (scaling, version control, Astronomer partnership) and product teams that write Dags, maintain metadata, and orchestrate ETL and MLOps.

04:00 Contrasting data problems at large ownership-first orgs (data reconciliation) versus mid-size lean-data orgs like Lyft (data engineers as the bottleneck).

07:04 Why Lyft treats Airflow as a service layer rather than a scheduled ETL system, including productionizing Jupyter notebooks from science teams.

10:14 The experience running Airflow on Astro, including the 2 to 3 upgrade and consolidated observability in one place.

12:13 The two problems driving self-service Dags: supporting 30+ users with a small data engineering team while meeting SOX compliance on driver pay changes.

14:38 How the GitHub Actions and Airflow integration works bidirectionally, blocking PRs until validation completes.

18:28 Why Lyft chose event-driven over polling: easier upstream-to-downstream context passing and avoiding wasted compute on on-demand work.

20:23 The two hardest production challenges: GitHub's tight coupling of check IDs to commit SHAs, and multi-hour simulations racing user rebases. How Lyft built an external context store to solve both.

25:15 Why Airflow 3.3's pluggable retry policy and task state store would unlock efficiency gains for transient GitHub failures and long-running Spark simulations.

26:39 Looking ahead to task loops and agentic operators for optimization problems with multiple correct answers.

Resources Mentioned:

• Lyft
• Apache Airflow
• Astro by Astronomer
• GitHub Actions

Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Automation #Airflow

Running Airflow with a one-person data engineering team means every decision about cost, tooling, and best practices has to earn its keep. In this episode, Kenten sits down with Aniruddha Sengupta, Staff Data Engineer at Insignis, to talk through how the UK cash-savings fintech runs a low-cost, scalable Airflow platform. The conversation covers observability, DAG-level best practices, the internal tools Insignis built to control costs, and where AI fits into the roadmap.

Key Takeaways:

00:00 Introduction.

02:22 Insignis data team as a "newspaper" for the business. Aniruddha describes the data team's role: consuming data from operational systems and external sources like Salesforce and Ringover, and delivering insight without owning business logic.

08:07 Airflow as the orchestrator, not the executor. Airflow sits at the heart of the platform, orchestrating SQL pipelines, API ingestion, and even acting as an observability layer over real-time pipelines built elsewhere.

12:19 Observability as definition of done. Why a green Airflow UI is not observability, and how Insignis builds independent dashboards to catch silently dropped data.

14:38 Choosing managed Airflow. How abstracting away the scheduler, workers, and database frees a small team to focus on code and documentation.

20:04 DAG patterns that keep costs down. CI/CD enforcement, failure callbacks, mindful DAG boundaries, and periodic purging of logs and metadata.

22:06 Tariff and Maestro. Two internal tools Insignis built: Tariff to surface SQL pipeline costs, and an AI-powered tool called Maestro to recommend worker, scheduler, and database sizing.

25:40 Exploring AI with Airflow. MLOps use cases around client behavior and the appeal of self-healing pipelines.

28:20 Wishlist for Airflow. Event-based DAG parsing so code changes surface in the UI closer to instantaneously.

Resources Mentioned:
Annual Airflow survey
- Insignis
- Apache Airflow
- Astronomer
- Building a Low-Cost Scalable Airflow Platform for Small Teams
- Google BigQuery

Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Automation #Airflow

Scaling Airflow to over 1,000 active DAGs on a team of 3-4 engineers takes structure. In this episode, Kenten Danas talks with Matt Stavinga, Data Engineer at Idelic (now part of Descartes), about how his team standardized DAG creation and logging, built a Jenkins-driven testing workflow with per-PR Airflow environments, and is opening up DAG configuration and triggering to their own customers through the Safety Suite UI.

Key Takeaways:

00:00 Introduction.

01:40 Idelic's mission: collecting data from North American trucking fleets and using it to identify risky drivers so fleets can intervene before accidents happen.

02:59 Where Airflow fits: Idelic runs on Astro, with each customer/integration combination as its own DAG.

03:56 The reality of vendor integrations: APIs, SFTP, email, screen scraping, and even executables installed on customer servers to push data out.

06:11 How the DAG count grew and where things started to hurt as a small team tried to keep up.

08:49 The standardized framework: abstract methods that wrap common Airflow functions and generate task groups for each ETL phase, plus standardized logging across vendors.

11:15 CI/CD with Jenkins: per-PR Airflow deployments that QA can spin up on demand through a Jira hook and tear back down when done.

13:31 Letting customers create their own DAGs through a wizard in Safety Suite, with plans for customer-triggered runs, filtered result review, and human-in-the-loop approval.

15:37 Using the Airflow API to trigger customer DAGs, and why Idelic pushes run data into its own database instead of relying on the Airflow logs API.

18:58 What Matt wants next from Airflow: task and DAG level trend tracking, and more from Otto on self-healing DAGs.

Resources Mentioned:

Airflow Survey: astronomer.typeform.com/airflowsurvey26

Astro by Astronomer: astronomer.io

Apache Airflow: airflow.apache.org

Otto (Astronomer's data engineering agent): https://www.astronomer.io/product/otto/

Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#ai #orchestration #airflow

Running LLMs over sensitive data means external APIs are off the table, but open-weight models still need reproducibility, validation, and a paper trail before downstream systems can depend on their output. In this episode, Kenten is joined by Chhayank Jain, Senior ML Engineer at Visa, to talk through how Airflow orchestrates local LLM pipelines end to end: nightly inference over millions of records, silent failure handling with Pydantic and distribution checks, and using a separate DAG to manage LoRA fine-tuning cycles.

Key Takeaways:

00:00 Introduction.

01:47 From modeling to pipelines. Chhayank shares how leading a readmission risk model across hospitals with different EHR systems turned him into a "pipeline person."

03:14 Day-to-day as an LLM engineer. Serving with ONNX Runtime, failure design, and agentic tooling with LangGraph. The model is only about one-fifth of the work.

05:03 The sensitive data problem. Why hosted LLM APIs are off the table for PHI and payments data, and why the real challenge is running Llama 3 nightly with enough checks that downstream models can trust the output.

07:34 Where Airflow fits. Sensors per site, dependency ordering, retries and backfills, and run records as the "paper trail" for audits.

10:03 DAG structure. Four stages: ingest and normalize, inference on GPU pods via the KubernetesPodOperator, validation, and load to the feature store.

12:32 Loud vs silent failures. OOM kills and timeouts are easy. Silent failures where the output looks clean but is wrong are the scary category, handled with Pydantic schemas, cross-checks against structured records, and distribution monitoring.

16:45 LoRA fine-tuning in its own DAG. A separate monthly or quarterly DAG pulls labeled data, fine-tunes with Hugging Face PEFT on top of Llama 3, and only writes a new adapter if it beats the frozen holdout.

20:33 Hardest production lessons. Pin everything (model version, temperature, features, seed), match parallelism to GPU capacity, and invest in traceability so you can answer "which adapter produced this row" six months later.

23:00 Airflow wishlist. First-class GPU awareness so teams stop reinventing it through pools and queues, and treating model versions as Airflow assets.

Resources Mentioned:

- Annual Airflow survey

- Airflow Summit talk: From Hours to Minutes: Orchestrating Local LLMs for Sensitive Data Pipelines with Apache Airflow

- Llama 3

- Hugging Face PEFT

- ONNX Runtime

- LangGraph

- Pydantic

Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Orchestration #Airflow

Security is one of the least visible parts of Airflow for most users, but it shapes what features teams can turn on in production. In this episode, Kenten Danas talks with Anish Giri, Senior Software Engineer and active Apache Airflow contributor focused on security, about three recent contributions that touch nearly every Airflow 3 deployment: hardening connection testing, adding JWT revocation, and splitting execution tokens by scope.

Key Takeaways:

00:00 Introduction.

01:16 Anish's inside-out path into Airflow, starting from source code and docs rather than running pipelines in production.

03:13 Why security engineering flips the usual software question from "what can this do?" to "what can anyone make this to?"

04:49 How connection testing worked before, why it was disabled in Airflow 2.7 after a CVE, and how the redesign moves execution off the API server.

06:26 Running connection tests on the worker in Airflow 3 so credentials never leave the API server boundary.

07:50 What the mailing list and architecture review looked like, including a first-round rejection and eventual consensus.

10:00 The JWT logout problem in Airflow 3 and the revocation table that makes logout an actual logout.

13:12 A production incident with a GPU pipeline task hitting 403 after 15 minutes, and why it required splitting the token in two.

15:41 How the workload token and the auth token now have different lifetimes and scopes, tied to the task queue timeout.

19:53 Advice for new Airflow contributors: start small, don't fear the PR review, and treat feedback as mentorship.

Resources Mentioned:

• Annual Airflow survey
• Apache Airflow
• Airflow on GitHub
• Airflow Slack
• Anish Giri at Airflow Summit

Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Orchestration #Airflow

Orchestrating data across more than 30 companies means most Airflow users on the platform aren't data engineers. In this episode, Kenten Danas talks with Lucas Trubiano, Data Engineer at itti, the technology company within Grupo Vázquez in Paraguay. Lucas walks through the custom YAML framework his Center of Excellence built on top of Airflow, how they baked data quality and custom operators into it, and how a spec-driven AI workflow now lets product and business users contribute to templates without knowing Python.

Key Takeaways:

00:00 Introduction.

01:47 What itti and Grupo Vázquez do, and the Data Engineering Center of Excellence's mandate to build a 360-degree view of the customer across more than 30 companies.

02:56 How Airflow fits in as a central task orchestrator (not a processing engine) across around 300 production DAGs.

04:25 Managing enterprise-scale Airflow: preferring Airflow-as-a-service, plus enabling self-service for non-technical users through YAML.

05:44 Why itti built a second, more opinionated YAML framework after DAG Factory-style customization created a code review bottleneck.

07:14 More than 80% of new DAGs are now created with the new framework because it's simply faster.

07:54 How the framework works end to end: Python DAGs, Jinja templates, YAML configs, and CI/CD compilation.

09:10 A Google Sheets ingestion example that shows how prevalidation, download, and processing tasks are hidden behind a simple YAML config to preserve reliability.

10:14 Building an in-house data quality tool that tests per partition instead of full-scanning tables, triggered via custom Airflow operators.

11:17 Custom operators for dbt, the in-house data quality tool, and AWS services like QuickSight dashboard refreshes, and how OSS Airflow makes them portable across instances.

15:35 Spec-driven development with a fork of GitHub Spec Kit so business users can describe what they want and let agents generate DAGs against certified templates.

17:24 Slack-native error routing: every DAG has an owner team, common errors ship with explanations, and only deep issues escalate to the central team.

19:29 Where they're heading next: agent-triggered pull requests for self-healing pipelines.

20:22 Airflow 3 wishlist: backfill improvements, event-driven orchestration for streaming pipelines, and Human in the Loop for generative AI DAGs.

Resources Mentioned:

• Apache Airflow
• Apache Spark
• DAG Factory
• GitHub Spec Kit

Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Automation #Airflow

Manufacturing data pipelines can't afford silent failures. When quality decisions and shop-floor visibility depend on Airflow, a broken DAG at midnight can cost real money. In this episode, Kenten Danas talks with Kumuda Sreenivasa, Founder of Receitly and Senior Data Architect at ATC Drivetrain, about how her team built self-healing Airflow pipelines, where AI fits into the recovery loop, and how they orchestrate AI agents as governed workflow components.

Key Takeaways:

00:00 Introduction.

02:29 How ATC Drivetrain uses Airflow across thousands of DAGs to orchestrate ETL/ELT jobs, data quality checks, and production reporting for a complex automotive remanufacturing environment.

04:08 Defining self-healing: pipelines that identify a known failure, decide whether they can recover safely, execute an approved action, and validate the result, all without paging an engineer.

07:03 The five-layer self-healing architecture: observe, classify, policy, recover, and validate.

10:00 Where AI fits in the recovery loop: classification, context gathering, and recommendations, but never bypassing operational policies or approval steps.

13:00 A concrete before/after: a currency-exchange failure caught overnight by AI-assisted recovery that saved four hours of downtime and roughly 120K.

14:27 Confidence levels and success rates: about 95% of small failure modes recover on their own.

15:34 AI-assisted troubleshooting at scale: how contextual log analysis and recommended actions save engineers from digging through thousands of log lines.

20:13 Orchestrating AI agents through Airflow: treating agents as bounded, governed workflow components with human approval and confidence-based stop conditions.

23:45 What Kumuda wants next from Airflow: stronger AI agent governance, standardized tracking of prompts and tool calls, and more flexible event-driven execution.

Resources Mentioned:

• Orchestrate Everything
• Apache Airflow
• ATC Drivetrain

Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Automation #Airflow

Strict sequential execution across DAGs sounds simple until you have scheduled pipelines and event-driven pipelines writing to the same MongoDB collections. Ivana Isailovic, Senior Big Data Engineer at Synechron, joins Marc Lamberti to walk through the three-layer DAG architecture her team built to solve exactly that, plus how they generate 200-task DAGs and how they rebuilt subdag-style group retries in Airflow 3.

Key Takeaways:

(00:00) Introduction.

(02:16) The stack: Snowflake source, MongoDB with a medallion (bronze, silver, gold) layout, Spark for processing, Airflow for orchestration, Elasticsearch for reports.

(05:29) Why standard Airflow options (max active runs, pools, dependency setups) each solved only part of the problem.

(06:40) Data consistency across bronze, silver, and gold layers is what forced strict sequential execution.

(09:11) Scheduled DAGs versus event-driven DAGs triggered at any moment from the application side.

(10:04) The three-layer architecture: trigger DAGs, a single proxy DAG that controls the queue, and main ETL DAGs.

(12:00) The queue is literally another DAG. The proxy DAG allows only one active run and serializes everything behind it.

(15:13) 200-task DAGs generated from nested task groups and YAML configuration files, with DAG versions tied to release numbers.

(17:38) How the layers talk to each other: sensors and TriggerDagRunOperator.

(19:44) Migrating from subdags to task groups without losing the ability to retry a whole group.

(21:49) Airflow 2.9 approach: reset task instance state via the metadata DB, keyed off the task group identifier.

(23:13) Airflow 3 approach: move the retry logic onto the official REST API for stability, security, and maintainability.

Resources Mentioned:

• Orchestrate Everything
• Apache Airflow
• Snowflake
• MongoDB
• Apache Spark
• Elasticsearch

Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Automation #Airflow

Financial data pipelines have to be right the first time. On this episode, Kenten sits down with [Antonello Benedetto](linkedin.com/in/anbento4), Staff Data Engineer at Wise, to talk about how the central data and analytics engineering team runs Airflow for critical financial datasets, tiers pipelines by reliability, and orchestrates LLM-enabled workflows with validation layers and agent-checking-agent patterns.

Key Takeaways:

• 00:00 Introduction.
• 01:47 What Wise does and Antonello's role in the central data and analytics engineering team, a hybrid platform-plus-analytics team that owns dbt infrastructure, BI, and Analytics MCPs as a service.
• 05:53 Three principles that guide Airflow pipeline design at Wise: a clean separation between orchestration and computation logic, computational awareness (offloading memory-intensive tasks to EMR or SageMaker), and standardized deployments.
• 07:21 Why Wise treats Airflow as a pure orchestration layer and pushes memory-intensive work to external workers.
• 08:45 Moving to the Python Virtual Environment Operator to standardize Airflow deployments across the org while giving analysts and data scientists per-job Python environments.
• 10:50 The tiering system for pipelines, how it distinguishes highly controlled, well-documented, well-observed pipelines from newer ones, and how requirements from downstream drive tier promotion.
• 17:18 Where LLM-enabled workflows differ from standard pipelines: validation layers for specific use cases, plus observability and evaluation platforms that track model performance across executions.
• 19:08 Using Airflow to orchestrate LLM generation of monthly variance commentary for analysts.
• 21:10 Handling non-idempotent LLM outputs with multi-layer validation against source-of-truth data, and using a second agent (CI/CD style) to validate the first agent's output.
• 23:04 How AI-enabled workflow orchestration differs from batch ETL, and why teams should start small before building fully agentic pipelines.
• 25:25 What Antonello would most like to see from Airflow next: native support for agentic workflows and better local development that mirrors production.

Resources Mentioned:

• [Orchestrate Everything](https://astronomer.link/data-flowcast-oe)
• [Wise](wise.com)
• [Wise Careers](wise.jobs)
• [Apache Airflow](airflow.apache.org)
• [dbt](getdbt.com)
• [Python Virtual Environment Operator](airflow.apache.org/docs/apache-airflow/stable/core-concepts/operators.html)

Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Automation #Airflow

Running a travel platform means dealing with fast-moving inventory, real-time fraud detection, and heavy performance marketing attribution, all while keeping data trustworthy for a data-driven org. In this episode, Mrinalini Singh, Data Platform Engineer at [Headout](headout.com), walks through how her team uses Airflow as the nervous system of their stack: orchestrating dbt with a write-audit-publish pattern, running ML training and inference, and wiring up alerting that points to the exact commit that broke a DAG.

Key Takeaways:

• 00:00 Introduction.
• 01:05 What a data platform engineer does at Headout, and the hub-and-spokes model where analysts and scientists write their own dbt models.
• 04:11 The specific data challenges of a travel platform: fast-changing inventory, real-time fraud analytics, and performance marketing attribution.
• 05:40 Where Airflow sits in the stack, from ingestion to transformation to serving.
• 07:01 The write-audit-publish dbt pattern and why slightly stale data beats wrong data.
• 09:55 Why Headout uses a custom Python operator instead of the dbt provider or Cosmos, reading the dbt manifest to build task groups per model.
• 13:45 ML use cases on Airflow: Feast feature store, model training, inference, and data/feature drift tracking.
• 17:00 Custom Slack failure hooks that stitch together Airflow logs, GitHub commit URLs, and teammate Slack IDs.
• 20:04 A zombie task incident that filled the metadata DB, caused locking issues, and drove the move to Grafana-based monitoring.
• 22:26 Using AI to generate Airflow code, encoding internal patterns as a skill file, and running an AI reviewer bot on every PR.

Resources Mentioned:

• [Orchestrate Everything](https://astronomer.link/data-flowcast-oe)
• [Headout](headout.com)
• [dbt](getdbt.com)
• [Cosmos](github.com/astronomer/astronomer-cosmos)
• [Feast](feast.dev)
• [Apache Flink](flink.apache.org)
• [Grafana](grafana.com)

Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Automation #Airflow