We are migrating our data lakehouse to Iceberg on Databricks. What are the current best practices for compaction and maintenance jobs on petabyte-scale tables?
Download run
Fan-out diagram
The searches the AI actually ran for this prompt.
SEED QUERY
We are migrating our data lakehouse to Iceberg on Databricks. What are the current best practices for compaction and maintenance jobs on petabyte-scale tables?
3 real searches · 1 real-search pass
Databricks Iceberg table maintenance OPTIMIZE compaction VACUUM snapshot expiration current documentation 2026Google ↗
OBSERVED
Databricks Iceberg managed tables predictive optimization maintenance compaction clustering documentationGoogle ↗
OBSERVED
Apache Iceberg maintenance rewrite_data_files rewrite_manifests expire_snapshots remove_orphan_files best practices petabyte scaleGoogle ↗
OBSERVED
Badge and edge weight = share of passes that produced the query. Hover a query for its intent and routing format.
Query classes
Google's semantic query class for every sub-query: what kind of answer each search is after, simulated vs real.
Classifying 3 queries…
Searches and sources
ChatGPT3 real searches · gpt-6
Databricks Iceberg table maintenance OPTIMIZE compaction VACUUM snapshot expiration current documentation 2026 Google ↗
Databricks Iceberg managed tables predictive optimization maintenance compaction clustering documentation Google ↗
Apache Iceberg maintenance rewrite_data_files rewrite_manifests expire_snapshots remove_orphan_files best practices petabyte scale Google ↗
- Predictive optimization for Unity Catalog managed tables · docs.databricks.com
- Unity Catalog managed tables for Delta Lake and Apache Iceberg · docs.databricks.com
- Maintenance · iceberg.apache.org
- Maintaining tables by using compaction · docs.aws.amazon.com
- How do I fix small-file problems in an Iceberg table? · answers.databricks.com
- Optimize data file layout · docs.databricks.com
- Procedures · iceberg.apache.org
- What are the best practices for compaction and maintenance on Delta or Iceberg? · answers.databricks.com
- Databricks Predictive Optimization · docs.databricks.com
- Apache Iceberg maintenance guide · iceberg.apache.org
Queries
3 queries · click a column to sort
| Query / type | Search | Intent | Query class | Routing format | Cluster | Freq ↓ | Weight |
|---|---|---|---|---|---|---|---|
Databricks Iceberg table maintenance OPTIMIZE compaction VACUUM snapshot expiration current documentation 2026 OBSERVED | Google ↗ AI Mode ↗ | — | — | — | — | 1x of 1 | |
Databricks Iceberg managed tables predictive optimization maintenance compaction clustering documentation OBSERVED | Google ↗ AI Mode ↗ | — | — | — | — | 1x of 1 | |
Apache Iceberg maintenance rewrite_data_files rewrite_manifests expire_snapshots remove_orphan_files best practices petabyte scale OBSERVED | Google ↗ AI Mode ↗ | — | — | — | — | 1x of 1 |