🧑 You
Tell me all about the $iceberg operator, I know it's the new hot thing in the analytics space (where new is now like 5+ years?)
🤖 Assistant
You’re right: Apache Iceberg is “new” in the modern-data-stack sense—established enough to be a standard architectural choice, but still rapidly evolving. MongoDB’s new piece is that Atlas Stream Processing (ASP) can now write directly to Iceberg tables on S3 using the $iceberg aggregation stage.
It reached GA on June 30, 2026.
$iceberg is a terminal Atlas Stream Processing stage—more precisely a pipeline stage, not a query operator. It continuously writes a stream into an Apache Iceberg table stored in an AWS S3 bucket.
Typical flow:
text
MongoDB change stream / Kafka / Kinesis
→ Atlas Stream Processing transforms or enriches events
→ $iceberg
→ Apache Iceberg tables in S3
→ Snowflake, Databricks, Spark, Trino, Athena, etc.
It gives you a direct operational-data-to-lakehouse path without having to assemble your own change-data-capture app, Kafka connector chain, or custom Iceberg writer.
Traditional data lakes store files—typically Parquet—on object storage. That is cheap and scalable, but a pile of files does not by itself provide database-like table behavior. Iceberg adds the table-management layer:
So instead of exporting MongoDB data as periodic CSV/JSON/Parquet dumps, you can maintain an analytical table in S3 that downstream engines understand as an Iceberg table. That makes it particularly relevant for real-time operational analytics, lakehouse architectures, and keeping an analytical copy of MongoDB data without treating an ETL pipeline as a bespoke software project.
javascript
{
$iceberg: {
connectionName: "myS3Connection",
bucket: "my-data-lake",
databaseName: "analytics",
tableName: "orders",
path: "iceberg-warehouse/",
region: "us-east-1", // required if processor is not on AWS
mode: "cdc", // "cdc" or "insert"; defaults to "cdc"
idFieldName: "_id", // optional; default "_id"
partitionedBy: {
_id: "identity",
orderDate: "day"
},
catalog: {
type: "glue" // "glue" or "hadoop"
}
}
}
It must be the last stage in the stream processor pipeline, and there can be only one terminal sink. In other words, a processor finishes with one of:
This is the default. ASP examines the streaming metadata field:
javascript
stream.source.operationType
and applies the corresponding mutation to the Iceberg table. That means an Atlas change stream can keep an Iceberg table aligned with source collection inserts, updates, replacements, and deletes. For example, this pattern mirrors a MongoDB collection to Iceberg:
javascript
const isDelete = {
$eq: [{ $meta: "stream.source.operationType" }, "delete"]
};
[
{
$source: {
connectionName: "atlasConn",
db: "commerce",
coll: "orders",
initialSync: { enable: true },
config: { fullDocument: "required" }
}
},
{
$match: {
operationType: { $in: ["insert", "update", "replace", "delete"] }
}
},
{
$replaceRoot: {
newRoot: {
$cond: {
if: isDelete,
then: "$documentKey",
else: "$fullDocument"
}
}
}
},
{
$iceberg: {
connectionName: "s3Lake",
bucket: "company-lake",
databaseName: "operational",
tableName: "orders",
path: "iceberg/",
mode: "cdc",
catalog: { type: "glue" }
}
}
]
initialSync seeds the existing collection, then the processor continues with ongoing change events.
This writes every input document as a new row and ignores operation type. Use it for things such as:
$iceberg requires:
It is not available on SP2 or SP5.
tableName may be a string expression, so one processor can route messages to distinct Iceberg tables:
javascript
tableName: { $meta: "stream.source.ns.coll" }
That is useful when a multi-collection source should land in separate analytical tables. Maximum dynamically routed tables per processor:
|
ASP tier |
Maximum tables |
|
SP10 |
5 |
|
SP30 |
10 |
|
SP50 |
50 |
This is a nice fit for MongoDB’s flexible schema, but it’s still prudent to apply a stable analytical contract upstream—e.g., normalize field names/types and avoid wild schema churn.
Output is at least once. That is consistent with ASP generally: downstream duplicates are possible after retries/restarts. Design around this when your consumer semantics require it—particularly for CDC operations and downstream derived tables.
javascript
catalog: { type: "hadoop" } // default-style file-based catalog
or:
javascript
catalog: { type: "glue" }
Use AWS Glue if you want the table catalog managed/discoverable in the broader AWS analytics ecosystem.
ASP maps common BSON types into Iceberg primitive types:
|
BSON |
Iceberg result |
|
string |
string |
|
int, long, double |
corresponding numeric type |
|
bool |
boolean |
|
date, timestamp |
UTC timestamptz, microsecond precision |
|
ObjectId |
hex string |
|
UUID |
string |
|
other BinData |
binary |
|
object / array |
Basic JSON string |
Important implication: MongoDB nested objects and arrays do not become nested Iceberg structs/lists through this stage; they are serialized as JSON strings. If analysts need first-class columns, transform/flatten the relevant data before $iceberg. Unsupported BSON types can go to the configured DLQ, so configure one for production pipelines.
You can define Iceberg partition transforms:
javascript
partitionedBy: {
_id: "identity",
eventDate: "day",
customerId: { bucket: 32 }
}
Supported transforms include:
The first partition transform must apply to the CDC identifier field (idFieldName, _id by default). Practical advice: use partitioning to help the analytical engines prune data, not merely to reproduce your MongoDB operational indexing strategy. Time-based partitioning is often sensible for event-like workloads; avoid extremely high-cardinality partitions unless there is a demonstrated query-planning benefit.
|
Need |
Use |
|
Write a current operational collection in MongoDB |
$merge |
|
Publish events to Kafka/Kinesis/PubSub, or land raw files in object storage |
$emit |
|
Maintain analytics/lakehouse tables in S3, queryable by Iceberg-aware engines |
$iceberg |
A useful mental model:
The big picture: $iceberg makes MongoDB a more direct producer for the lakehouse world. Iceberg itself is no longer “brand-new,” but the operational-DB-to-open-table-format path is increasingly the modern alternative to forcing every analytics workload through a proprietary warehouse copy or a fragile hand-assembled CDC pipeline.