Skip to main content

Export lineage as OpenLineage to Amazon S3

Setup takes about 20 minutes.

Introduction

Foundational can export lineage to an Amazon S3 bucket that teams own, as OpenLineage events. Any catalog or observability tool that reads OpenLineage from cloud storage can then pick it up, without needing a dedicated connector.

To make the connection, you need to:

  • Configure the OpenLineage export destination
    ​

  • Deploy the CloudFormation stack in AWS
    ​

  • Connect to Foundational
    ​

  • Read the export

Foundational exports

Foundational writes lineage to the configured bucket on a recurring schedule. Each export replaces the file with the organization's full, current lineage. Treat the file as a snapshot, not an append-only log.

What each export contains

Element

Details

Format

OpenLineage RunEvents, one JSON object per line (newline-delimited JSON)

File

A single file, openlineage-events.jsonl, directly under the prefix you configure, or at the bucket root if the prefix is left empty. There are no sub-folders.

Contents

Table-level lineage (each job with its input and output datasets); column-level lineage in the columnLineage facet; and, where Foundational has them, table and column descriptions in the documentation facet plus Foundational's AI-generated explanations of how columns are transformed

When exports run

Behavior

Details

Interval setting

Select an Export Interval during setup.. The minimum and the default are both 14 days.

First export

Foundational starts a first export as soon as the connector is saved.

Subsequent exports

Foundational writes a new export once the interval has passed since the last successful one.

Failed export

Foundational retries the next day rather than waiting until the next scheduled date.

Status visibility

The connector's card on the Integrations page shows the time and status of the last export.

Prerequisites

Ensure you have:

  • AWS IAM permissions to create roles and deploy CloudFormation stacks.
    ​

  • An Amazon S3 bucket to hold the export, and the AWS Region it is in.
    ​

  • Added Foundational IP addresses to the allowlist. See the article Allowing IP access to Foundational.

Access permissions

The CloudFormation template creates an IAM role that:

  • Trusts only Foundational's AWS account, and only when it presents the external ID.
    ​

  • Can use the S3 bucket, scoped to the bucket and to objects under the configured prefix.

Required permissions

Service

Required Permissions

Resources

Amazon S3

s3:PutObject, s3:GetObject, s3:DeleteObject, s3:ListBucket, s3:GetBucketLocation

The configured bucket, and objects under the configured prefix.

Foundational does not read or modify anything else in the AWS account.

Create the OpenLineage export

The export has four parts:

  • Configure the OpenLineage export destination
    ​

  • Deploy the CloudFormation stack in AWS
    ​

  • Connect to Foundational
    ​

  • Read the export

Part 1: Configure the OpenLineage export destination

  1. In Foundational, open the Integrations page.
    ​

  2. In Catalogs, select the OpenLineage card and click Connect.
    ​

  3. A new screen opens. Click Add Account.
    ​

  4. The initial setup screen opens. Click Start Setup.
    ​

  5. The Set Up New Connection screen opens.
    ​

    Enter the details:

    • S3 Bucket Name: The bucket, without s3://.
      ​

    • Prefix (optional): The folder within the bucket, if any. For example, foundational/openlineage, which keeps the export apart from anything else in the bucket.
      ​

    • AWS Account ID: The account that owns the bucket. The CloudFormation stack in Part 2 creates the role in that account.
      ​

    • IAM Role Name: The name the CloudFormation template will give the role. Foundational generates the template with this name.
      ​

    • External ID: Auto-generated, and read-only. The template locks the role's trust policy to this value. Copy and save it. You’ll need it to deploy the CloudFormation stack in Part 2.
      ​

    • Export Interval (days): How often Foundational re-exports. Minimum and default is 14.
      ​

  6. Click Next.
    ​

  7. A new screen opens. Click Download YAML File.
    ​

  8. Do not click Save. Leave the Foundational screen open for now. You'll return to this screen in Part 3.

Part 2: Deploy the CloudFormation stack in AWS

This part uses the auto-generated External ID saved in Part 1.

  1. In AWS, go to the CloudFormation Stacks page.
    ​

  2. In Specify template, select Upload a template file and upload the YAML file downloaded in Part 1.
    ​

  3. Click Next.
    ​

  4. In Provide a stack name, enter a stack name of your choice (e.g., FoundationalOpenLineageExport).
    ​

  5. Click Next.
    ​

  6. In Capabilities, select: I acknowledge that AWS CloudFormation might create IAM resources with custom names.
    ​

  7. Click Next.
    ​

  8. At the bottom of the page, click Submit.
    ​

  9. The stack creation process takes about a minute. Track the status on the CloudFormation stacks page.
    ​

  10. When the process completes, move to Part 3.

Part 3: Connect to Foundational

  1. In Foundational, return to the screen you left open.
    ​

  2. In the final setup screen, click Save to complete the connection.

    Note: Before it stores the configuration, Foundational checks it can actually write: it assumes the role, writes a small probe object (.foundational-export-writecheck) under the configured prefix, and then deletes it. If the check fails, the error appears here. Correct the field and save again.

    If the bucket enforces SSE-KMS encryption with a customer-managed key, also allow the role kms:GenerateDataKey on that key.

Foundational is now connected to OpenLineage. To read exports, continue to Part 4.

Part 4: Read the export

Downstream tools need their own read access to the bucket.

Grant the tool access to:

  • s3:GetObject
    ​

  • s3:ListBucket
    ​

  • s3:GetBucketLocation if the tool looks up the bucket's region

Point the tool at:

  • Bucket: the bucket name, without s3://
    ​

  • Path / prefix: the configured prefix, or empty for the bucket root.
    ​

  • File mask: *.jsonl

Schedule the tool to read at least as often as the Export Interval, so no export goes unread. More frequent reads are harmless; they simply find the same file.

That's it. The exported lineage is ready to read.

Troubleshooting

Review the following issues if the connection fails or an export doesn't behave as expected.

Saving fails with "Could not assume the trust role…"

Issue

What's wrong

What to do

Saving fails with "Could not assume the trust role…"

The AWS account ID, role name, or external ID doesn't match the CloudFormation stack.

Check that you created the stack in the account entered, using the downloaded template.

Saving fails with "Could not write to s3://…"

Issue

What's wrong

What to do

Saving fails with "Could not write to s3://…"

Foundational assumed the role, but the bucket rejected the write.

Check the bucket name, and that the role's policy covers the bucket and prefix. For SSE-KMS buckets, also check the key permission.

The card shows the export failed

Issue

What's wrong

What to do

The card shows the export failed.

The error message appears on the card.

Foundational retries the export on the next nightly run.

The downstream tool finds no files

Confirm that its path and *.jsonl mask match the prefix, and that the connector card shows at least one successful export.

Related Articles
Did this answer your question?