SageMaker Data Lineage Now Supports IAM Domains

SageMaker data lineage finally supports IAM-based domains, but don't expect a full audit trail.

What changed

• Amazon SageMaker Unified Studio now captures lineage events from Apache Spark jobs (EMR, Glue, SageMaker Visual ETL, notebooks) in IAM-based domains.
• This feature was previously limited to IAM Identity Center-based domains.
• An interactive lineage graph with configurable depth and column-level detail is available.
• Programmatic publishing, querying, and management of lineage events via OpenLineage APIs are supported.

Why it matters

This update enables organizations using IAM for domain access control to gain visibility into their SageMaker data pipelines. The honest version: Previously, if your organization mandated IAM-based domains for security or compliance, you couldn't easily track data movement and transformations within SageMaker. This change allows for better debugging, auditing, and understanding of data flow for workloads like Apache Spark jobs running on EMR or Glue, directly within SageMaker Studio. It pairs with, and extends the utility of, services like AWS Glue Data Catalog for a more complete data governance picture.

The catch

Watch out: While the announcement mentions