migrating-hive-tables-to-s3-tables
Skill3.5k starsChanged 4 days agoApache-2.0View on GitHub
Migrates a Glue Data Catalog Hive table (Parquet) into Amazon S3 Tables as managed Iceberg using the native S3 Tables migration API (StartTableMigration, SyncTableMigration, CompleteTableMigration), a managed service call, not a self-run CTAS, Glue, or EMR job. The source is a Hive table; an Iceberg source is never in scope, even when the destination is S3 Tables. Covers plan review, migrate, sync, complete, and validate. Same-account, same-region, Glue Hive Parquet source only. Use when the request asks to migrate or import a Hive table into S3 Tables, or names the native migration API. Not applicable to converting a self-managed or standard Iceberg table into S3 Tables (ingesting-into-data-lake covers that), self-run CTAS/Glue-ETL/Spark migrations (ingesting-into-data-lake), or creating empty tables (creating-data-lake-table).
Adopted by
Steady: 0 new projects took it in the last 4 weeks, none the 4 weeks before.thin data
Who's talking about aws/agent-toolkit-for-aws
- A Hacker News story“Agent Toolkit for AWS”Hacker News · Jun 8, 2026 · 5 points
- @yug1224.com“AIエージェントがAWS環境の構築や運用を効率よく行うための公式ツールキットらしい ClaudeやCursorなどのエージェントに導入して開発を自動化するのが楽しみ yug1224 starred …”Bluesky · Jun 27, 2026 · 2 likes
- @librecanada.bsky.social“GitHub - Webpage "GitHub - aws/agent-toolkit-for-aws: Official, AWS-supported MCP servers, skills, and …”Bluesky · May 10, 2026 · 2 likes
- @yug1224.com“AIエージェントにAWSの操作や開発をサポートさせるためのツールキットらしい 実務でAIを活用してインフラ構築を効率化するのに使えそう yug1224 starred aws/agent-toolkit-for-aws…”Bluesky · Jun 27, 2026 · 1 likes
- sudosubin/agents.nix“kiro-powers.github.aws.agent-toolkit-for-aws: 0-unstable-2026-11-14 → 0-unstable-2026-11-15”GitHub · Nov 15, 2026
The biggest 5 of 55 mentions linking to the repository that our crawl found. Each line is their own headline.
See all 55
Talked about on GitHub, Bluesky and Hacker News (55 times).
On GitHub · 50
- Nov 15, 2026kiro-powers.github.aws.agent-toolkit-for-aws: 0-unstable-2026-11-14 → 0-unstable-2026-11-15 — sudosubin/agents.nix0 reactions
- Nov 15, 2026agent-skills.github.aws.agent-toolkit-for-aws: 0-unstable-2026-11-14 → 0-unstable-2026-11-15 — sudosubin/agents.nix0 reactions
- Nov 15, 2026codex-marketplaces.github.aws.agent-toolkit-for-aws: 0-unstable-2026-11-14 → 0-unstable-2026-11-15 — sudosubin/agents.nix0 reactions
- Nov 15, 2026codex-plugins.github.aws.agent-toolkit-for-aws: 0-unstable-2026-11-14 → 0-unstable-2026-11-15 — sudosubin/agents.nix0 reactions
- Nov 15, 2026claude-code-marketplaces.github.aws.agent-toolkit-for-aws: 0-unstable-2026-11-14 → 0-unstable-2026-11-15 — sudosubin/agents.nix0 reactions
45 more on GitHub
- Nov 15, 2026agent-plugins.github.aws.agent-toolkit-for-aws: 0-unstable-2026-11-14 → 0-unstable-2026-11-15 — sudosubin/agents.nix0 reactions
- Nov 15, 2026claude-code-plugins.github.aws.agent-toolkit-for-aws: 0-unstable-2026-11-14 → 0-unstable-2026-11-15 — sudosubin/agents.nix0 reactions
- Nov 15, 2026copilot-plugins.github.aws.agent-toolkit-for-aws: 0-unstable-2026-11-14 → 0-unstable-2026-11-15 — sudosubin/agents.nix0 reactions
- Nov 15, 2026agent-skills.github.aws.agent-toolkit-for-aws: 5955ad2 → 9ca16b9 — sudosubin/agents.nix0 reactions
- Nov 15, 2026codex-marketplaces.github.aws.agent-toolkit-for-aws: 5955ad2 → 9ca16b9 — sudosubin/agents.nix0 reactions
- Nov 15, 2026codex-plugins.github.aws.agent-toolkit-for-aws: 5955ad2 → 9ca16b9 — sudosubin/agents.nix0 reactions
- Nov 15, 2026claude-code-marketplaces.github.aws.agent-toolkit-for-aws: 5955ad2 → 9ca16b9 — sudosubin/agents.nix0 reactions
- Nov 14, 2026kiro-powers.github.aws.agent-toolkit-for-aws: 5955ad2 → 9ca16b9 — sudosubin/agents.nix0 reactions
- Nov 14, 2026agent-plugins.github.aws.agent-toolkit-for-aws: 5955ad2 → 9ca16b9 — sudosubin/agents.nix0 reactions
- Nov 14, 2026agent-skills.github.aws.agent-toolkit-for-aws: 0-unstable-2026-11-13 → 0-unstable-2026-11-14 — sudosubin/agents.nix0 reactions
- Nov 14, 2026codex-marketplaces.github.aws.agent-toolkit-for-aws: 0-unstable-2026-11-13 → 0-unstable-2026-11-14 — sudosubin/agents.nix0 reactions
- Nov 14, 2026codex-plugins.github.aws.agent-toolkit-for-aws: 0-unstable-2026-11-13 → 0-unstable-2026-11-14 — sudosubin/agents.nix0 reactions
- Nov 14, 2026claude-code-marketplaces.github.aws.agent-toolkit-for-aws: 0-unstable-2026-11-13 → 0-unstable-2026-11-14 — sudosubin/agents.nix0 reactions
- Nov 14, 2026kiro-powers.github.aws.agent-toolkit-for-aws: 0-unstable-2026-11-13 → 0-unstable-2026-11-14 — sudosubin/agents.nix0 reactions
- Nov 14, 2026claude-code-plugins.github.aws.agent-toolkit-for-aws: 0-unstable-2026-11-13 → 0-unstable-2026-11-14 — sudosubin/agents.nix0 reactions
- Nov 14, 2026agent-plugins.github.aws.agent-toolkit-for-aws: 0-unstable-2026-11-13 → 0-unstable-2026-11-14 — sudosubin/agents.nix0 reactions
- Nov 14, 2026copilot-plugins.github.aws.agent-toolkit-for-aws: 0-unstable-2026-11-13 → 0-unstable-2026-11-14 — sudosubin/agents.nix0 reactions
- Nov 14, 2026claude-code-marketplaces.github.aws.agent-toolkit-for-aws: 905d95c → 19b89b2 — sudosubin/agents.nix0 reactions
- Nov 14, 2026kiro-powers.github.aws.agent-toolkit-for-aws: 905d95c → 19b89b2 — sudosubin/agents.nix0 reactions
- Nov 14, 2026agent-plugins.github.aws.agent-toolkit-for-aws: 94c87f7 → 19b89b2 — sudosubin/agents.nix0 reactions
- Nov 14, 2026claude-code-plugins.github.aws.agent-toolkit-for-aws: 94c87f7 → 19b89b2 — sudosubin/agents.nix0 reactions
- Nov 14, 2026copilot-plugins.github.aws.agent-toolkit-for-aws: 94c87f7 → 19b89b2 — sudosubin/agents.nix0 reactions
- Nov 14, 2026codex-marketplaces.github.aws.agent-toolkit-for-aws: 94c87f7 → 19b89b2 — sudosubin/agents.nix0 reactions
- Nov 13, 2026agent-skills.github.aws.agent-toolkit-for-aws: 94c87f7 → 19b89b2 — sudosubin/agents.nix0 reactions
- Nov 13, 2026claude-code-marketplaces.github.aws.agent-toolkit-for-aws: c48a8a8 → 905d95c — sudosubin/agents.nix0 reactions
- Nov 13, 2026kiro-powers.github.aws.agent-toolkit-for-aws: c48a8a8 → 905d95c — sudosubin/agents.nix0 reactions
- Nov 13, 2026claude-code-plugins.github.aws.agent-toolkit-for-aws: 73575fe → 94c87f7 — sudosubin/agents.nix0 reactions
- Nov 13, 2026agent-plugins.github.aws.agent-toolkit-for-aws: 73575fe → 94c87f7 — sudosubin/agents.nix0 reactions
- Nov 13, 2026copilot-plugins.github.aws.agent-toolkit-for-aws: 73575fe → 94c87f7 — sudosubin/agents.nix0 reactions
- Nov 13, 2026codex-marketplaces.github.aws.agent-toolkit-for-aws: c48a8a8 → 94c87f7 — sudosubin/agents.nix0 reactions
- Nov 13, 2026codex-plugins.github.aws.agent-toolkit-for-aws: 73575fe → 19b89b2 — sudosubin/agents.nix0 reactions
- Nov 13, 2026agent-skills.github.aws.agent-toolkit-for-aws: 73575fe → 94c87f7 — sudosubin/agents.nix0 reactions
- Nov 13, 2026claude-code-marketplaces.github.aws.agent-toolkit-for-aws: 73575fe → c48a8a8 — sudosubin/agents.nix0 reactions
- Nov 13, 2026kiro-powers.github.aws.agent-toolkit-for-aws: 73575fe → c48a8a8 — sudosubin/agents.nix0 reactions
- Nov 13, 2026copilot-plugins.github.aws.agent-toolkit-for-aws: 0-unstable-2026-11-12 → 0-unstable-2026-11-13 — sudosubin/agents.nix0 reactions
- Nov 13, 2026codex-marketplaces.github.aws.agent-toolkit-for-aws: 0-unstable-2026-11-12 → 0-unstable-2026-11-13 — sudosubin/agents.nix0 reactions
- Nov 13, 2026codex-plugins.github.aws.agent-toolkit-for-aws: 0-unstable-2026-11-12 → 0-unstable-2026-11-13 — sudosubin/agents.nix0 reactions
- Nov 13, 2026agent-skills.github.aws.agent-toolkit-for-aws: 0-unstable-2026-11-12 → 0-unstable-2026-11-13 — sudosubin/agents.nix0 reactions
- Nov 13, 2026claude-code-marketplaces.github.aws.agent-toolkit-for-aws: 0-unstable-2026-11-12 → 0-unstable-2026-11-13 — sudosubin/agents.nix0 reactions
- Nov 13, 2026kiro-powers.github.aws.agent-toolkit-for-aws: 0-unstable-2026-11-12 → 0-unstable-2026-11-13 — sudosubin/agents.nix0 reactions
- Nov 13, 2026agent-plugins.github.aws.agent-toolkit-for-aws: 0-unstable-2026-11-12 → 0-unstable-2026-11-13 — sudosubin/agents.nix0 reactions
- Nov 13, 2026claude-code-plugins.github.aws.agent-toolkit-for-aws: 0-unstable-2026-11-12 → 0-unstable-2026-11-13 — sudosubin/agents.nix0 reactions
- Nov 12, 2026agent-skills.github.aws.agent-toolkit-for-aws: 7ab0d27 → 3bc7d13 — sudosubin/agents.nix0 reactions
- Nov 12, 2026Opt-in WAF request logging — theJuddson/terraform-aws-devopsfoundry0 reactions
- Nov 12, 2026Confused-deputy conditions on service role trust policies — theJuddson/terraform-aws-devopsfoundry0 reactions
On Bluesky · 4
- Jun 27, 2026AIエージェントがAWS環境の構築や運用を効率よく行うための公式ツールキットらしい ClaudeやCursorなどのエージェントに導入して開発を自動化するのが楽しみ yug1224 starred aws/agent-toolkit-for-aws… — @yug1224.com2 likes and reposts · 1 reply
- May 10, 2026GitHub - Webpage "GitHub - aws/agent-toolkit-for-aws: Official, AWS-supported MCP servers, skills, and plugins to help AI agents build on… — @librecanada.bsky.social2 likes and reposts
- Jun 27, 2026AIエージェントにAWSの操作や開発をサポートさせるためのツールキットらしい 実務でAIを活用してインフラ構築を効率化するのに使えそう yug1224 starred aws/agent-toolkit-for-aws… — @yug1224.com1 likes and repost
- May 8, 2026aws/agent-toolkit-for-aws: Official, AWS-supported MCP servers, skills, and plugins to help AI agents build on AWS #devopsish — @chrisshort.net0 likes and reposts · 1 reply
On Hacker News · 1
- Jun 8, 2026Agent Toolkit for AWS5 points
In this repository203 other files it gives its agents.
- analyzing-release-readinessSkill
- chatting-with-aws-devops-agentSkill
- coordinating-multi-space-devops-agentSkill
- diff-scanning-with-aws-security-agentSkill
- investigating-incidents-with-aws-devops-agentSkill
- pentesting-with-aws-security-agentSkill
- remediating-with-aws-security-agentSkill
- running-release-testsSkill
- scanning-with-aws-security-agentSkill
- setup-devops-agentSkill
- setup-security-agentSkill
- setupSkill
- threat-modeling-with-aws-security-agentSkill
- agents-buildSkill
- agents-connectSkill
- agents-debugSkill
- agents-deploySkill
- agents-get-startedSkill
- agents-hardenSkill
- agents-optimizeSkill
- agents-paySkill
- agents-paySkill
- amazon-bedrockSkill
- aws-ai-mlSkill
- aws-authSkill
- aws-billing-and-cost-managementSkill
- aws-blocksSkill
- aws-cdkSkill
- aws-cloudformationSkill
- aws-computeSkill
- aws-containersSkill
- aws-databaseSkill
- aws-deploymentSkill
- aws-emulatorSkill
- aws-iamSkill
- aws-investigationSkill
- aws-messaging-and-streamingSkill
- aws-networkingSkill
- aws-observabilitySkill
- aws-sdk-js-v3-usageSkill
- aws-sdk-python-usageSkill
- aws-sdk-swift-usageSkill
- aws-secrets-managerSkill
- aws-securitySkill
- aws-serverlessSkill
- aws-storageSkill
- aws-well-architected-reviewSkill
- launch-with-awsSkill
- signing-in-to-awsSkill
- amazon-opensearch-serviceSkill
- authoring-mwaa-workflowSkill
- connecting-to-data-sourceSkill
- creating-data-lake-tableSkill
- debugging-mwaa-workflowSkill
- developing-applications-on-managed-service-for-apache-flinkSkill
- exploring-data-catalogSkill
- finding-data-lake-assetsSkill
- ingesting-into-data-lakeSkill
- managing-amazon-mskSkill
- managing-data-asset-accessSkill
- The first 60 shown.
The file
Show its rules and text
We split this skill into the rules it gives an agent
13 rules. About 0 are common, 4 are distinctive, and 9 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈4
Distinctive · estimate
This author's own choices. Worth a look.
9
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 32
When the AWS MCP Server is available, use it to load this skill and its references (aws___retrieve_skill, with file="references/<name>.md") …
- Distinctive: line 56
Every writer to the destination writes through an Iceberg-aware engine (for example Athena, EMR or Glue with Iceberg).
- Distinctive: line 58
Every reader supports Iceberg and the S3 Tables catalog.
- Very specific: line 74
Use this skill when the source is a Glue Data Catalog Hive table with Parquet data (TableType=EXTERNAL_TABLE, Hive Parquet SerDe) and the …
- Very specific: line 116
All-or-nothing copy: if a single object fails, the whole migration fails.
- Very specific: line 153
Do not call start-table-migration until all of these pass.
- Very specific: line 159
Partition locations: Every partition location is under the table location.
- Distinctive: line 160
Lake Formation: If the source table or its location is registered with Lake Formation, stop: not supported (confirm current support in the …
- Very specific: line 164
KMS: If data is encrypted with a customer managed key, grant kms:Decrypt and kms:GenerateDataKey.
- Very specific: line 170
Run checks 1 to 7 on the succeeded table before Complete, because Sync is refused after Complete.
- Very specific: line 177
Schema review: every column's type label (Lossless, Widened, Converted) was accepted in the plan.
- Very specific: line 186
Do not add sts:ExternalId to the trust policy.
- Distinctive: line 198
No secrets in generated commands, reports, or state files.
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Migrating Hive Tables to S3 Tables
Contents
- Overview
- Before you migrate
- When to use
- When NOT to use (use another skill)
- Scope and constraints
- The 6 migration APIs
- Migration lifecycle
- Pre-migration guardrails
- Post-migration guardrails
- Key API shapes
- Security considerations
- References
Overview
This skill guides an agent through migrating one Glue Hive Parquet table into an S3 table bucket with the native S3 Tables migration API. Amazon S3 Tables plans the migration, reports column and object compatibility, copies the data, and keeps the destination in sync with the source until you complete it. Your coding agent, following this skill, runs preflight checks, starts the migration, presents the plan for approval, monitors progress, completes the migration, and validates the result.
When the AWS MCP Server is available, use it to load this skill and its references
(aws___retrieve_skill, with file="references/<name>.md") and to run the AWS calls.
Before you migrate
Agent instruction: You MUST present the migration plan and obtain explicit user approval
before any step that changes tables, data, metadata, or catalog entries, and before any IAM, KMS,
or S3 bucket change (creating or editing a role or policy, editing a key policy, creating a report
bucket). Show the exact JSON or command first. For a key policy, read it with get-key-policy,
add one statement, and never replace the whole policy. The plan MUST list:
- the source table and the destination table bucket, namespace, and table name;
- the write paths that will change (ETL jobs, streaming or ingest,
INSERT/MERGE, CDC); - the query engines and downstream consumers that read the table;
- the datasets, saved queries, and permissions that reference the old Glue table or S3 prefix.
Up-front approval in the user's request (naming the source, destination, and intent to migrate)
covers only read-only checks and a Start with plan review. Approving the plan, accepting any
setNull or drop policy, Complete, and any delete always need explicit approval during the
conversation. If the actual state differs from what was approved (a different schema,
partition layout, row count, or resource), stop, explain the difference, and get approval again.
Confirm these constraints with the user as part of the plan:
- Every writer to the destination writes through an Iceberg-aware engine (for example Athena, EMR or Glue with Iceberg). Writers that write Parquet files directly to S3 will not work.
- Every reader supports Iceberg and the S3 Tables catalog.
- Jobs and queries move from the Glue Hive table to the S3 Tables catalog reference.
- The source is not changed. Keep writers on the source until the destination is validated.
- Validate row counts and sample values (references/validation.md) before moving production traffic.
See Migrating tables to S3 Tables for the full list of supported sources and limits.
Runtime: works with or without the AWS MCP server; steps use AWS CLI syntax. The AWS MCP server is recommended for sandboxed execution and audit logging. The migration commands are part of
aws s3tables; if an older CLI lacks them, see references/api-setup.md.
When to use
Use this skill when the source is a Glue Data Catalog Hive table with Parquet data
(TableType=EXTERNAL_TABLE, Hive Parquet SerDe) and the destination is an S3 table bucket in the
same account and Region:
- Migrating that table into S3 Tables as managed Iceberg with the native migration API.
- Keeping the migrated table in sync with the active Hive source before cutover.
When NOT to use (use another skill)
The names below are other agent skills. For these requests, do not use this skill; load the named skill instead. When asked when not to use this skill, list every case:
- A self-run CTAS, Glue ETL, or EMR/Spark migration into Iceberg: use the
ingesting-into-data-lakeskill (this skill covers only the managed migration API). - Converting a self-managed or standard Iceberg table: use the
ingesting-into-data-lakeskill. - Loading raw files (Parquet with no Glue table, CSV, JSON) or other sources (JDBC, Snowflake, BigQuery, DynamoDB): use the
ingesting-into-data-lakeskill. - Creating a new empty Iceberg table: use the
creating-data-lake-tableskill. - Running or managing Athena queries: use the
querying-data-lakeskill. - Locating tables or assets: use the
finding-data-lake-assetsskill. Auditing the catalog: use theexploring-data-catalogskill. - Setting up Glue connections: use the
connecting-to-data-sourceskill. - Any destination other than S3 Tables.
Prompt patterns (trigger phrasings)
- "Migrate my Glue Hive table
<db>.<tbl>into my table bucket<name>." - "Import an existing Hive table into S3 Tables using the native migration API."
- "Move my Parquet Hive table to Iceberg without running CTAS or a Glue/Spark job."
- "Use StartTableMigration to convert
<tbl>." - "Keep my migrated table in sync with the source before cutover" → Sync.
- "Complete my migrated table" → Complete.
- "Check the status or plan of migration
<id>" → Get. - "List my table migrations" → List.
Scope and constraints
- Source: Glue Hive Parquet table in the same account and region as the destination table bucket.
- Destination: an existing S3 table bucket and namespace; the migration creates the table.
- Not supported (confirm the current list in the S3 Tables migration guide): Lake Formation registered tables, partitions outside the table location, non-Parquet formats, Iceberg sources, cross-account, SSE-C, and archived storage classes. Check these in preflight and stop if any apply. For a Lake Formation registered table, tell the user it is not supported and suggest contacting AWS Support or their account team with the migration ID to request support. Never change Lake Formation settings.
- Compatibility problems are reported during planning. Always review the plan before approving.
- All-or-nothing copy: if a single object fails, the whole migration fails. A
succeededrun has no failed objects. - Three reports are written to a general-purpose bucket you supply: table compatibility, object compatibility, and object migration.
- Cost and duration: the migration writes a second copy of the data; plan for roughly double storage until the source is decommissioned. Large tables can take hours.
- Quotas: see S3 Tables quotas.
- Migration policies: by default an incompatible column fails the run.
incompatibleValuesPolicy: setNullandunmatchedSourceFieldsPolicy: dropapply to the whole column and the run still reportssucceeded. Treat anysetNullordroppedcolumn as data loss that needs explicit approval.
The 6 migration APIs
| API | Purpose | Phase |
StartTableMigration | Plan and begin a migration. With --review-migration-plan, pauses at pendingReview. | Start |
GetTableMigration | Returns state, hasValidPlan, planningStats, migrationStats, migrationOutput, failureReason. | Track, review |
UpdateTableMigration | Approve (--requested-state queued) or cancel. | Approve |
SyncTableMigration | Apply source changes since the last plan; repeat until current. | Sync |
CompleteTableMigration | Promote the destination to read-write. Takes --table-arn. Returns an empty body; the migration state stays succeeded. Confirm with get-table (managedByService is null). | Complete |
ListTableMigrations | List migrations for one table bucket or table. | Discover |
Request and response shapes are in references/lifecycle.md.
Migration lifecycle (phases)
- Preflight → references/preflight.md, references/iam-roles.md
- Start and plan:
start-table-migration --review-migration-plan;planning→pendingReview. - Review:
get-table-migration; read the compatibility reports and present the plan for approval. - Approve and migrate:
update-table-migration --requested-state queued; pollmigrating→succeeded. - Sync (optional):
sync-table-migration; repeat until current. - Validate: on the
succeededtable, before Complete, query the destination in Athena and compare with the source → references/validation.md. Validation is part of every migration: run the row-count and sample checks without being asked, unless the user opts out. If counts are short, reconcile withsync-table-migration(not possible after Complete). - Complete: once parity holds and the user approves,
complete-table-migration --table-arn ..., then confirmget-tableshowsmanagedByServicenull.
Phases 2 to 7 are in references/lifecycle.md.
Pre-migration guardrails (run every time, before Start)
Do not call start-table-migration until all of these pass. Commands are in
references/preflight.md and references/iam-roles.md.
- Plan approved. The plan in Before you migrate has been presented and approved.
- Credentials and region.
sts get-caller-identitysucceeds; the source and destination are in the same account and region. - Source is Glue Hive Parquet.
TableType=EXTERNAL_TABLE, Hive Parquet format and SerDe; not Iceberg, not ACID/transactional, no storage handler. RecordcatalogId. - Partition locations. Every partition location is under the table location; partitions outside it fail planning with
UNSUPPORTED_PARTITIONS. Check for partition projection andskip.header.line.count. - Lake Formation. If the source table or its location is registered with Lake Formation, stop: not supported (confirm current support in the migration guide). Do not change Lake Formation settings.
- Destination exists. The table bucket and namespace exist. A missing namespace fails Start with
NotFoundException. - Migration role trust. One statement, principal
migration.s3tables.amazonaws.com, conditionsaws:SourceAccountandaws:SourceArn. - Migration role permissions. Scoped source S3 and Glue read, destination write, report bucket write. The caller also needs
iam:PassRoleon the role, conditioned oniam:PassedToService=migration.s3tables.amazonaws.com. - KMS. If data is encrypted with a customer managed key, grant
kms:Decryptandkms:GenerateDataKey. For an SSE-KMS destination, the key policy must also allowmaintenance.s3tables.amazonaws.com, or table creation fails and planning never completes (see preflight §7 and IAM roles).
Post-migration guardrails (verify success before Complete)
Full detail in references/validation.md.
Run checks 1 to 7 on the succeeded table before Complete, because Sync is refused after Complete. After Complete, only confirm that get-table shows managedByService null.
- Terminal state:
state: succeeded,failureReasonnull. - Object accounting:
objectsFailedMigrationis 0 andobjectsMigratedmatches the planned object count. If it differs, check the source object count (an extra file from an earlier sync is legitimate) and confirm with row-count parity before treating it as a failure. - Row count parity: destination
count(*)equals source. - Partition parity: per-partition row counts match (partitioned tables).
- Spot-check rows: sample values match, including converted columns such as timestamps.
- Schema review: every column's type label (
Lossless,Widened,Converted) was accepted in the plan; none isIncompatibleorUnsupported. - Catalog naming: query as
"s3tablescatalog/<table-bucket>"."<namespace>"."<table>".
Key API shapes
sourceType=glueTable(top level) andsource={"glueTable":{"catalogId":"<acct>","databaseName":"<db>","tableName":"<tbl>"}}.catalogIdis required. The role member isroleARN.--destination-table={"tableBucketARN":"<arn>","namespace":"<ns>","name":"<tbl>"}.migrationOutputis required:{bucket, prefixes}, wherebucketis a general-purpose bucket ARN andprefixesoptionally setstableCompatibilityReport,objectCompatibilityReport,objectMigrationReport.- Optional
dataMigration.migrationPolicies:unmatchedSourceFieldsPolicy(fail|drop),incompatibleValuesPolicy(fail|setNull). Both defaultfail. - Do not add
sts:ExternalIdto the trust policy; Amazon S3 Tables does not send one. - States:
new → planning → pendingReview → queued → migrating → succeeded, plusfailed,cancelling,cancelled. - On cancel or failure the destination table is kept. If
hasValidPlanis true, resume withsync-table-migration; otherwise delete the table and start again. Rules in references/lifecycle.md. - Complete only a
succeededmigration. Completing twice returns409 ConflictExceptionor400 BadRequestException; treat either as "already done." - Athena reference:
"s3tablescatalog/<table-bucket>"."<namespace>"."<table>". Requires the table bucket to be integrated with the Glue Data Catalog.
Security considerations
- Instructions only; no shipped code or dependencies.
- Migration role: trusts
migration.s3tables.amazonaws.comwithaws:SourceAccountandaws:SourceArnconditions; permissions scoped to the exact source, destination, and report resources. - Encryption: SSE-KMS on the destination table bucket and report bucket is recommended, with TLS-only bucket policies. A customer managed destination key must allow
maintenance.s3tables.amazonaws.comin its key policy. - Logging: enable CloudTrail for the migration APIs and role assumptions, S3 data-event or server-access logging on the buckets involved, and CloudWatch alarms on failures. Encrypt log destinations.
- No secrets in generated commands, reports, or state files.
References
| File | Topic |
| references/api-setup.md | Confirming the migration commands are in the aws s3tables CLI |
| references/preflight.md | Credentials, source checks, partition locations, Lake Formation, destination |
| references/iam-roles.md | Caller policy and migration role trust and permissions |
| references/lifecycle.md | State machine and per-phase calls: start, review, approve, sync, complete |
| references/validation.md | Athena row-count, aggregate, and sample checks; catalog naming |
| references/troubleshooting.md | failureReason codes and fixes |
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

