Skip to main content

How to Export Lineage to Collibra

Setup takes about 30–45 minutes.

Introduction

Connecting Collibra to Foundational allows Foundational to export its computed lineage into your Collibra data catalog, so your team can explore Foundational's table- and column-level lineage alongside your existing Collibra assets.

The connection process has three parts:

  • Part 1: Configure the export in Foundational
    ​

  • Part 2: Configure Collibra Edge to read the bucket
    ​

  • Part 3: Run the sync, schedule it, and verify

How it works

  1. Foundational exports lineage: Foundational writes your full organization lineage (schema, table-level, and column-level) as OpenLineage run events to an S3 bucket you own, on the Export Interval you configure.
    ​

  2. Collibra Edge connects to the bucket: Your Collibra Edge site reads the same bucket through an AWS connection you configure (Part 2).
    ​

  3. The OpenLineage capability ingests the files: Collibra's OpenLineage (Cloud) capability parses the exported files and creates or updates lineage in your catalog.
    ​

  4. A recurring sync keeps lineage current: Collibra re-ingests on the schedule you set, picking up each new export as Foundational produces it.
    ​

  5. Lineage attaches to your catalog assets by System: Collibra links a lineage node to a catalog asset only when their full paths match: System › database › schema › table. The capability's Source configuration (Part 2, Step 2) tells Collibra which System each source in the exported files belongs to. Foundational generates this configuration for you.

Foundational exports as newline-delimited OpenLineage JSON files (.jsonl), OpenLineage spec 1-0-0. The integration is one-way: Foundational only writes files to the bucket and never reads from your catalog.

Prerequisites

Ensure you have:

  • Added Foundational IP addresses to the allowlist, see the article Allowing IP access to Foundational.
    ​

  • Administrator access to the AWS account that will host the destination S3 bucket, and the ability to create an S3 bucket and an IAM role.
    ​

  • The AWS Region for the bucket.
    ​

  • A Collibra Edge site in Healthy status (self-hosted or a Collibra-managed Cloud site), permission to add Edge connections and capabilities, and the Technical Lineage global permission (needed to view lineage and its status page).
    ​

  • (Optional) The Catalog assets destination set up, if you also want Database, Schema, Table, and Column assets in your catalog for this lineage to attach to. See the article How to Export Catalog Assets to Collibra.

Note: This integration uses Collibra's OpenLineage (Cloud) technical-lineage capability, which reads files from cloud storage (S3). The local Shared Storage variant is not used.

Access permissions

The role Foundational assumes needs the following permissions on your bucket and prefix:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": ["s3:PutObject", "s3:GetObject", "s3:DeleteObject", "s3:ListBucket", "s3:GetBucketLocation"],
    "Resource": [
      "arn:aws:s3:::<bucket>",
      "arn:aws:s3:::<bucket>/<prefix>/*"
    ]
  }]
}

This allows Foundational to:

  • Write the exported lineage files to your bucket and prefix
    ​

  • List objects in the bucket, to confirm the destination exists and is reachable
    ​

  • Delete the temporary object it writes during setup, to verify write access (s3:DeleteObject)
    ​

  • Resolve the bucket's AWS Region before writing (s3:GetBucketLocation)

Note: Foundational does not read from or modify anything else in your AWS account. The CloudFormation template in Part 1 creates a role with this policy and the correct trust relationship. You don't need to build it by hand.

Part 1: Configure the export in Foundational

  1. In your AWS account, create an S3 bucket (or choose a dedicated prefix within an existing bucket) to hold the exported files, for example s3://acme-collibra-lineage/foundational. Note the bucket name, the optional prefix, and the Region.
    ​

  2. In Foundational, open the Connectors & Integrations page. In Catalogs, select the Collibra card.
    ​

  3. Click Add Destination.
    ​

  4. In Choose an export destination, on the Lineage to S3 card, click Set up
    ​


    The wizard has three steps: Configure Collibra Export, Download CloudFormation YAML, and Configure in Collibra.
    ​

  5. In Configure Collibra Export, complete the connection details:
    ​

    • S3 Bucket Name: Your bucket, without s3://.
      ​

    • Prefix (optional): The folder within the bucket, if any.
      ​

    • AWS Account ID: The account that owns the bucket.
      ​

    • IAM Role Name: The name of the role the CloudFormation template creates (e.g., FoundationalLineageExport). Enter the name you intend to use. The template is generated from it.
      ​

    • External ID: Generated for you and read-only. The template locks the role's trust policy to this value.
      ​

    • Export Interval (days): How often Foundational re-exports. Minimum and default is 14.
      ​

  6. Click Next. A new screen opens.​

  7. Click Download YAML File. The template is generated from the values you entered in step 5. Leave Foundational open. You return to it in step 9.
    ​

  8. In AWS, go to CloudFormation > Create stack, choose Upload a template file, and upload the downloaded YAML.
    ​
    Give the stack a name and finish creating it. This creates the cross-account IAM role that trusts Foundational, with the permissions described in Access permissions above.
    ​

  9. Back in Foundational, click Save.
    ​

    Before it stores the configuration, Foundational assumes the new role and writes a temporary object to the bucket to confirm access.

    If the check fails, the error appears in the wizard. Use the Edit button on the previous step to correct the field, then click Save again.

    Warning: Download the YAML and save it in the same browser session. Foundational regenerates the External ID if you reload the form, so a role created from an older template no longer matches what Foundational sends when it assumes the role.

  10. After you save, the wizard opens Configure in Collibra.

This step lists the Collibra-side steps from Parts 2 and 3. It also includes a Source configuration generated for your organization: a JSON document that lists every namespace Foundational exports your tables under.
​
11. Copy the Source configuration. You paste it into the capability in Part 2, Step 2.

Note: If the step says Foundational is still listing your namespaces, reopen it in a few minutes to get the full list. </aside>

12. Click Done to leave the wizard.
​
13. Once you save the connector, it shows as configured. Its card displays the last export time and status after the first export completes.

Part 2: Configure Collibra Edge to read the bucket

To configure Collibra Edge to read the bucket, you need to:

  • Add an AWS connection (step 1)
    ​

  • Add the OpenLineage capability (step 2)
    ​

Step 1: Add an AWS connection

There are two authentication methods:

  • Use Option A if your Edge runs on an AWS EC2 instance whose instance role can read the bucket.
    ​

  • Use Option B if you authenticate with a dedicated IAM principal.
    ​

Option A: EC2 instance role

  1. Go to Settings > Edge > [your site] > Connections > Add connection > AWS connection.
    ​

  2. For Authentication Type, select EC2.
    ​

  3. Confirm the EC2 instance's IMDS hop limit is 2.

The AWS default is 1. The harvester runs in pods and can't fetch role credentials at that value.
4. Save the connection.

Note: The principal needs s3:Get* and s3:List* on the bucket and its objects. Plain GetObject/ListBucket is not enough. The harvester also calls, e.g., GetBucketLocation.

Option B: IAM access key

  1. Go to Settings > Edge > [your site] > Connections > Add connection > AWS connection.
    ​

  2. For Authentication Type, select IAM.
    ​

  3. Enter the Access Key ID and Secret Access Key for a principal that can read the bucket.
    ​

  4. Click Save.

    Note: The principal needs s3:Get* and s3:List* on the bucket and its objects. Plain GetObject/ListBucket is not enough. The harvester also calls, e.g., GetBucketLocation.

Step 2: Add the OpenLineage capability

  1. Go to Settings > Edge > [your site] > Capabilities > Add capability.
    ​


    ​

  2. Choose Technical Lineage for OpenLineage (Cloud).
    ​

  3. Complete the capability fields:
    ​

    • Name: Any name (e.g., foundational-openlineage)
      ​

    • Description: Free text
      ​

    • Source ID: A unique source name, with no / or special characters (e.g., foundational_lineage)
      ​

    • Cloud Connection: The AWS connection you just created
      ​

    • Cloud Storage Bucket/Container: Your bucket name (no s3://)
      ​

    • Cloud Storage Region: e.g., eu-west-1
      ​

    • Cloud Storage path: Your prefix (leave empty for the bucket root)
      ​

    • Mask: *.jsonl
      ​

    • Processing Level: Sync for a full ingest (use Analyze first to preview)
      ​

    • Properties: Leave empty. Only on Collibra Platform for Government, add techlinHost and techlinKey (your Collibra Data Lineage service's host and API key, from your Collibra account team). Collibra requires them only there.
      ​

    • Source configuration: Paste the JSON from the last step of the Foundational wizard (Part 1, step 10).
      ​

      The Source configuration has two parts:
      ​

      • datasources: Maps each namespace Foundational exports (for example, snowflake://<account>) to a Collibra System (collibraSystemName). Foundational fills in Collibra's default System name for each one: the namespace with every / replaced by _. Rename a System to match the System your catalog already files that database under, so the lineage attaches to your existing assets. If you use the Catalog assets destination, Foundational reads this mapping on every export and creates its Databases under the same Systems.
        ​

      • customNamespaceParsers: Tells Collibra how to read Foundational's Snowflake and Postgres namespaces, which its built-in patterns don't recognize. Keep it as is.
        ​

      The following example uses placeholders. The real configuration comes from the wizard and lists your own namespaces:
      ​

      {
        "customNamespaceParsers": [
          {
            "type": "database",
            "namespacePattern": "^postgres://(?P<host>.+)$",
            "nameTemplate": "{database}.{schema}.{table}",
            "dialect": "postgres"
          },
          {
            "type": "database",
            "namespacePattern": "^snowflake://(?P<host>.+)$",
            "nameTemplate": "{database}.{schema}.{table}",
            "dialect": "snowflake"
          }
        ],
        "datasources": [
          {
            "collibraSystemName": "snowflake:__<account>",
            "type": "database",
            "namespace": "snowflake://<account>",
            "dialect": "snowflake"
          },
          {
            "collibraSystemName": "postgres:__<host>",
            "type": "database",
            "namespace": "postgres://<host>",
            "dialect": "postgres"
          }
        ]
      }

  4. Click Add.

Part 3: Run the sync, schedule it, and verify

  1. Go to Catalog > Integrations > Integration Configuration. Find the connection you created in Part 2 and click your capability in the Capabilities column.
    ​

  2. In Synchronization Configuration, click Synchronize. A notification confirms the run started, and its status shows on this capability's page.

    Warning: A "Completed" status does not guarantee Collibra ingested the lineage. Collibra silently skips events with problems (for example, a dataset namespace it doesn't recognize), and the run still reports success. Always verify (step 6).

  3. On the same capability page, in the Synchronization Schedule section, click Add Schedule.
    ​

  4. Set Repeat to Daily (or use a Cron expression, Quartz format), an At time (on the hour only, e.g., 14:00), and your Time zone.
    ​

  5. Click Save.

    Note: Run the sync at least as often as your Export Interval, so no export goes un-ingested. A daily schedule is a safe default. Syncing more often than Foundational exports is harmless, since a run that finds no new files simply does nothing.

  6. Open any Table or Column asset page and click the Technical Lineage tab.
    ​
    In the Browse pane, your source appears under the System you mapped its namespace to, with the database, schema, table, and column tree below it. The graph shows table-level and column-level edges.

    A table whose lineage links to its catalog asset opens its own node in the graph. If it doesn't, check that the Source configuration includes its namespace and maps it to the System its database belongs to in your catalog.
    ​

  7. Check for skipped events. Click View status (the 📈 icon, bottom-right). The Transformations table lists per-event results for your Source ID. ANALYZE_ERROR entries explain any dropped event

    Note: If the sync finds no files, check that the bucket, prefix, and *.jsonl mask match Part 1 exactly, and that at least one export has completed.

That's it. Foundational is now connected to Collibra.

Validation and troubleshooting

Lineage appears under a System such as snowflake:__<account> instead of attaching to your existing tables

Symptom

Cause / fix

Lineage appears under a System such as snowflake:__<account> instead of attaching to your existing tables

The Source configuration doesn't map the namespace to the System your catalog files that database under. In the capability's Source configuration, set that namespace's collibraSystemName to your System's exact name. Then run Synchronize again.

Events are dropped even though the sync reports success

Symptom

Cause / fix

Events are dropped even though the sync reports success

Namespaces are load-bearing. Collibra maps each dataset namespace to a source type using the OpenLineage naming convention and silently drops non-conforming events.
​
Check the View status > Transformations table for ANALYZE_ERROR details. If ANALYZE_ERROR says a Snowflake or Postgres dataset "does not match any supported dataset type", check that the Source configuration includes the customNamespaceParsers from Foundational.


Collibra can't import some source types as datasets, such as Kafka topics, MongoDB collections, REST APIs, and mainframe files. Collibra skips events for those types, and the rest of the lineage still imports.

"Catalog authorization failed", or the lineage viewer hangs on "Loading Collibra"

Symptom

Cause / fix

"Catalog authorization failed", or the lineage viewer hangs on "Loading Collibra"

This points to the Collibra Data Lineage service or entitlement on your instance, not your AWS configuration. Contact Collibra support.
​
Only Collibra Platform for Government instances need the techlinHost and techlinKey capability properties, to point at the correct regional lineage service. See Part 2, Step 2.

Self-hosted Edge site won't install or run the harvester

Symptom

Cause / fix

Self-hosted Edge site won't install or run the harvester

A self-hosted Edge site needs at least 16 vCPU / 64 GB (enforced by the installer), RHEL with the k3s SELinux policy, nm-cloud-setup disabled, and IMDS hop limit 2 on EC2. See Collibra's Edge installation documentation.

Useful links

Related Articles
Did this answer your question?