agentleFS
Google CloudKnown publisherGoogleCloudPlatform/composer-utilities/cicd/gemini_fixes/.agents/skills/airflow-best-practices/SKILL.md

airflow-best-practices

Skill11 starsChanged 3 months agoApache-2.0View on GitHub

Optimizes Airflow DAG code following established operational, performance, and maintainability best practices.

1 other project uses a version of this.

Adopted by

Most recently taken up by

  • Other versions1 project uses a version of this.

    Word for word · 0

    None found.

    Edited · 0

    None found.

    Wrote their own version · 1

    1. baekenough/oh-my-customcode★ 42same name

    How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.

  • In this repositoryOne other file it gives its agents.

The file

We split this skill into the rules it gives an agent

9 rules. About 0 are common, 5 are distinctive, and 4 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈5

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 4

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Check: Ensure top-level code is strictly limited to DAG definitions, basic configuration dicts, and imports.

    line 13
  2. Very specific:

    Action: Always fetch Variables and Connections within the task execution (e.g., inside the Python callable for a PythonOperator).

    line 17
  3. Distinctive:

    Rule: Every DAG and Task should produce the exact same deterministic outcome no matter how many times it runs.

    line 20
  4. Distinctive:

    Ensure database writes are UPSERTs rather than basic INSERTs if rows might already exist.

    line 22
  5. Very specific:

    Rule: Start dates should always be fixed and static (e.g., datetime(2023, 1, 1) or pendulum.datetime(2023, 1, 1, tz="UTC")).

    line 26
  6. Very specific:

    Action: Ensure every DAG has a comprehensive default_args definition that sets standard retries (e.g., 2 or 3) and retry_delay (e.g., …

    line 35
  7. Very specific:

    Action: When working with Python callables and standard PythonOperator, refactor to use the @task decorator where it improves readability …

    line 39
  8. Distinctive:

    When a user asks you to optimize or review a DAG:

    line 46
  9. Distinctive:

    Add comments in the refactored code explaining to the user why certain values were moved into an execution context or changed to use Jinja …

    line 50

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Airflow DAG Code Optimization Skill

You are an expert in Apache Airflow. Your goal is to review and optimize Airflow DAGs to ensure they are high-performing, maintainable, and conform to best practices for scalable Airflow environments (such as Google Cloud Composer). When analyzing or refactoring DAG code, apply the following guidelines and refactoring patterns while making sure to print your step-by-step reasoning to Cloud Build via standard output so that they're visible in the Cloud Build logs:

1. Top-Level Code Constraints

  • Rule: DAG files are parsed entirely on every scheduler heartbeat.
  • Action: Move all heavy computations, network calls (APIs), and database queries outside of the top-level Python scope and into the task operators or execution contexts.
  • Check: Ensure top-level code is strictly limited to DAG definitions, basic configuration dicts, and imports.

2. Connections and Variables Management

  • Rule: Accessing Variable.get() or Connection.get() at the top layer issues a database hit every time the file is parsed by the scheduler, severely degrading Airflow performance.
  • Action: Always fetch Variables and Connections within the task execution (e.g., inside the Python callable for a PythonOperator). Alternatively, use Jinja templating {{ var.value.my_var }} inside templated arguments where native operators support it.

3. Idempotency Check

  • Rule: Every DAG and Task should produce the exact same deterministic outcome no matter how many times it runs.
  • Action:
    • Ensure database writes are UPSERTs rather than basic INSERTs if rows might already exist.
    • Data writes to storage should overwrite existing partitions, or write into execution-specific locations using {{ ds }}.

4. Start Dates and Execution Time

  • Rule: Start dates should always be fixed and static (e.g., datetime(2023, 1, 1) or pendulum.datetime(2023, 1, 1, tz="UTC")). Using dynamic dates like datetime.now() causes the scheduler to continuously shift the start time and tasks may never trigger.
  • Action: Replace any dynamic start_date with a fixed, static date. Also, avoid using catchup=True unless specifically requested by the user.

5. XCom Usage

  • Rule: XComs are stored in the Airflow Metadata DB and are intended only for small metadata. Passing large datasets (like Pandas Dataframes) via XCom brings down the Airflow database and hurts scheduler performance.
  • Action: Refactor tasks that pass heavy payload data via XCom to instead write the data to external storage (GCS/S3) and only pass the resulting URI or path via XCom.

6. Default Arguments and Retries

  • Rule: Tasks should gracefully handle transient failures.
  • Action: Ensure every DAG has a comprehensive default_args definition that sets standard retries (e.g., 2 or 3) and retry_delay (e.g., timedelta(minutes=5)).

7. TaskFlow API Adoption

  • Rule: Modern Airflow execution should leverage the TaskFlow API (@task, @dag) for cleaner, Pythonic data passing.
  • Action: When working with Python callables and standard PythonOperator, refactor to use the @task decorator where it improves readability and makes explicit data dependencies obvious.

8. Granular and Atomic Tasks

  • Rule: "One task, one logical operation."
  • Action: Split monolithic tasks that handle extraction, transformation, AND loading into distinct atomic tasks. If a task fails, it must be retriable without recreating side-effects for work already finished.

Workflow for DAG Optimization

When a user asks you to optimize or review a DAG:

  1. Scan the code and identify anti-patterns (e.g., top-level DB calls, dynamic start dates, heavy XCom usage).
  2. Clearly explain each identified issue to the user and reference the best practice rule violated.
  3. Perform an in-place replacement of the DAG files with the optimized DAG code containing the necessary corrections.
  4. Add comments in the refactored code explaining to the user why certain values were moved into an execution context or changed to use Jinja templating.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.