agentleFS
Sign inStart free
gordonmurraygordonmurray/data-engineering-skills/paimon/SKILL.md

paimon

Skill47 starsChanged 4 months agoMITView on GitHub

Design, ingest into, tune, and operate Apache Paimon tables for streaming lakehouses and multimodal data. Use for Paimon primary-key or append-only table design, bucket sizing, changelog producer choice, Flink CDC ingestion, compaction backlog, lookup joins, PyPaimon, blob and vector storage, Spark reads, or Iceberg compatibility.

The file

We split this skill into the rules it gives an agent

7 rules. About 0 are common, 6 are distinctive, and 1 is very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈6

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 1

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Release authority: use signed ASF distributions and matching Maven/PyPI artifacts.

    line 25
  2. Distinctive:

    Run compaction as a dedicated job for high-volume streaming tables so compaction cannot backpressure ingestion.

    line 58
  3. Distinctive:

    Use lookup cache only when the dimension table fits in memory and the staleness it introduces is acceptable.

    line 60
  4. Distinctive:

    Do not drop and recreate a table to resolve a schema problem that schema evolution can handle.

    line 70
  5. Distinctive:

    Keep REST catalog and object storage credentials out of table properties and out of SQL committed to the repository.

    line 72
  6. Distinctive:

    After an ingestion change, confirm the snapshot count is advancing and the commit interval matches expectation.

    line 77
  7. Very specific:

    After compaction or bucket changes, compare file count and average file size per bucket via $files.

    line 79

≈ “Common” is an estimate: at least 1 in 5 of the 350,495 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Apache Paimon Expert

Scope

Paimon table design, Flink-native streaming ingestion, changelog semantics, compaction, lookup joins, Spark reads, Iceberg compatibility, and Paimon 2.x multimodal tables and Python integrations.

For Flink job architecture and operations use the flink skill. For Iceberg table internals use iceberg; this skill covers Paimon's Iceberg compatibility mode, not Iceberg itself.

Current Facts

Release guidance checked on 2026-11-03. Recheck the linked sources before selecting versions for a new deployment; preserve existing project pins.

  • Current stable Paimon and PyPaimon: 2.1.0, published October 12, 2026. Paimon 2.0 shipped in August; it is no longer a roadmap-only release.
  • Release authority: use signed ASF distributions and matching Maven/PyPI artifacts. Paimon and PyPaimon share a release version and vote; do not infer availability from a development branch or GitHub Releases alone.
  • Python: install pypaimon; its source lives in apache/paimon. Verify optional native accelerators and their versions separately from the Python SDK.
  • Flink integration: 2.1.0 engine artifacts cover Flink 1.16–1.20 and 2.0–2.2. paimon-flink-2.3:2.1.0 is not published. CDC 3.7.0 distributes for 1.20.x and 2.2.x, so choose their intersection for CDC pipelines.
  • Spark integration: 2.1.0 publishes Spark 4.1 and 4.0 artifacts with Scala 2.13, and Spark 3.2–3.5 artifacts with Scala 2.12. Spark 4.1 is no longer master-only.
  • 2.x scope: the released documentation covers data evolution, blob and vector storage, global indexes, full-text search, and PyPaimon integration with Ray and PyTorch. Check the specific table mode and engine before applying primary-key streaming advice to a multimodal table.

Inspect First

Establish before recommending or changing anything:

  1. Table type, bucket mode, and bucket count, from the DDL or DESCRIBE. Primary-key and append-only tables behave differently under every subsequent decision.
  2. Paimon, Flink, and Flink CDC versions actually in use.
  3. The current changelog producer setting, and whether any downstream consumer reads changelog at all.
  4. For performance work, read the $files, $snapshots, $manifests, and $options system tables. Get file count per bucket, average file size, compaction backlog, and snapshot expiry settings rather than assuming.
  5. For a 1.x to 2.x upgrade, inspect the target release's migration notes and each reader/writer dependency. For multimodal work, identify blob storage, index type, and the Python/native components actually installed.

Decision Rules

  • Include partition fields in the primary key when the table is partitioned.
  • Size buckets up front. Changing the bucket count of a fixed-bucket table requires rewriting existing data, so choose against expected volume rather than accepting the default.
  • Choose the changelog producer from real downstream need: none when nothing consumes changelog, input when the source already emits a complete changelog, and lookup or full-compaction when it must be generated. The last two carry real write-side cost.
  • Run compaction as a dedicated job for high-volume streaming tables so compaction cannot backpressure ingestion.
  • Use lookup cache only when the dimension table fits in memory and the staleness it introduces is acceptable.

Safety

  • Changing bucket count, primary key, or partition spec on an existing table requires a data rewrite. State the data volume and expected duration before proposing it.
  • Snapshot expiry deletes files that time travel and lagging streaming consumers still need. Check consumer lag before shortening retention.
  • Do not drop and recreate a table to resolve a schema problem that schema evolution can handle; recreating discards snapshot history.
  • Keep REST catalog and object storage credentials out of table properties and out of SQL committed to the repository.

Verify

  • After an ingestion change, confirm the snapshot count is advancing and the commit interval matches expectation.
  • After compaction or bucket changes, compare file count and average file size per bucket via $files.
  • For CDC pipelines, check row counts and a sample of updated and deleted keys against the source. A running job is not evidence of correct output.
  • Report the Paimon and Flink versions and which system tables you read.

Update Checklist

  • Recheck Apache Paimon tags and signed ASF distributions, then confirm Maven artifacts and PyPI pypaimon publication for the selected release.
  • Recheck Flink CDC compatibility for the selected Flink and Paimon releases.

Sources

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

Check your setup, free