Airflow DAG Run Orchestrator
detail.loadingPreview
Automates Airflow DAG runs, monitors their status, and handles retries or errors.
About This Workflow
This workflow orchestrates the execution of Apache Airflow DAGs. It initiates a DAG run, then continuously polls Airflow for its status. Based on the state (queued, running, success, failed), it either waits for completion, retrieves results, or stops the process with an appropriate error message.
The workflow is designed to be robust by implementing a retry mechanism with a configurable wait time and a maximum attempt count.
Key Features
- Trigger Airflow DAG runs programmatically.
- Monitor DAG run status in real-time.
- Implement wait-and-retry logic for queued or running states.
- Define maximum wait times to prevent infinite loops.
- Retrieve XCom values upon successful task completion.
- Gracefully handle failed DAG runs by stopping execution.
How To Use
-
**Input Data (
in datanode):dag_id: The ID of the Airflow DAG to run.task_id: The ID of the task whose return value you want to retrieve.conf: A JSON string representing the configuration for the DAG run.wait(optional): The initial waiting time in seconds between status checks (defaults to 10).wait_time(optional): The maximum number of retries/checks before considering the DAG run as timed out (defaults to 12).
-
**Airflow API Configuration (
airflow-apinode):- Set the
prefixto your Airflow API endpoint (e.g.,https://airflow.example.com).
- Set the
-
**Credentials:
- Ensure your Airflow HTTP Basic Auth credentials are set up correctly in n8n.
-
**Execution Flow:
- The
Airflow: dag_runnode starts a new DAG run. - The
if state == queuednode checks if the DAG is queued. If so, it waits (Waitnode) and re-checks. - If the DAG is not queued (or after waiting),
Airflow: dag_run - statefetches the current status. - The
Switch: statenode directs the flow based on the DAG run's state:success: Proceeds toAirflow: dag_run - get resultto fetch XComs.queuedorrunning: Goes back to thecountnode to increment retry count and then to theWaitnode.failed: Triggersdag run fail.
- The
countnode tracks the number of status checks. If count > wait_timechecks if the maximum retry limit has been reached. If so, it triggersdag run wait too long; otherwise, it continues to wait.
- The
Apps Used
Workflow JSON
{
"id": "0464e082-180e-4e64-940e-24574e7e48cb",
"name": "Airflow DAG Run Orchestrator",
"nodes": 16,
"category": "Orchestration",
"status": "active",
"version": "1.0.0"
}Note: This is a sample preview. The full workflow JSON contains node configurations, credentials placeholders, and execution logic.
Get This Workflow
ID: 0464e082-180e...
About the Author
Free n8n Workflows Official
System Admin
The official repository for verified enterprise-grade workflows.
Statistics
Related Workflows
Discover more workflows you might like
Universal CSV to JSON API Converter
Effortlessly transform CSV data into structured JSON with this versatile n8n workflow. Integrate it into any application as a custom API endpoint, supporting various input methods including file uploads and raw text.
Automate Local Business Outreach with AI-Powered Yelp Scraper
This workflow automates the process of scraping local business details from Yelp using AI, then leverages that data to send personalized partnership proposals via Gmail. It's perfect for sales and marketing teams looking to streamline lead generation and outreach campaigns.
Automate Getty Images Editorial Search & CMS Integration
This n8n workflow automates searching for editorial images on Getty Images, extracts key details and embed codes, and prepares them for seamless integration into your Content Management System (CMS), streamlining your content creation process.