Setup takes about 15–20 minutes.
Introduction
Connecting Databricks to Foundational allows Foundational to read Unity Catalog metadata and build lineage across your Databricks workspaces.
To make the connection, you need to:
Step 1: Create credentials
Step 2: Give Foundational a SQL warehouse
Step 3: Grant access (metastore admin)
Step 4 (recommended): Show query text (account admin)
Step 5 (optional): Turn on notebook lineage
Prerequisites
Ensure you have:
Unity Catalog enabled on the workspace.
Added Foundational IP addresses to the allowlist. See the article Allowing IP access to Foundational.
Access to a workspace admin, a metastore admin (step 3) and an account admin (step 4, recommended).
Access permissions
Foundational connects to Databricks to:
Read Unity Catalog metadata: catalogs, schemas, tables, columns, comments, primary and foreign keys, owners, tags, row filters, column masks and table type.
Build lineage from Unity Catalog metadata, primary and foreign keys, and AI/BI dashboards. To add notebook and job lineage, turn on the Notebook lineage option (step 5).
Show tables in a foreign catalog that mirrors Snowflake (Lakehouse Federation) as the Snowflake table they mirror, and keep their Databricks name as an alias.
Run the monitors and data-contract checks you set up on Databricks tables, on your SQL warehouse.
All access is read-only. Foundational reads metadata, system tables and table commit history. It never scans your table data, except for monitors and data-contract checks you configure yourself. Each scan reads table statistics from table metadata. This runs your SQL warehouse briefly once a day.
Coming soon
Features planned for future releases | Grants it uses |
Track how each table is used: reads, writes, deletes and failed queries. |
|
Track freshness and volume: when data last changed (ignoring maintenance such as OPTIMIZE and VACUUM), table size, and row count when Databricks can answer it from table metadata. | Catalog |
Show cost: DBUs and dollars per warehouse, job, pipeline, user and table, plus an estimate of the cloud machine cost of classic clusters. |
|
What each grant enables
Grant | What it enables |
Workspace access + Databricks SQL access entitlements | Required to connect. |
Can use on the warehouse | Every SQL-based read below. Without it, catalog metadata only. |
USE CATALOG, USE SCHEMA, SELECT on catalogs | Primary and foreign keys. Table statistics: freshness, size and row count (coming soon). Without it, names and columns only. |
| Unity Catalog table and column lineage, and discovery of your other workspaces. Read and write tracking (coming soon). |
| The SQL that produces each table (with Step 4). Query statistics and failed queries (coming soon). |
| Cost in DBUs and dollars (coming soon). |
| The warehouse cost estimate on the connector card. Warehouse and cluster cost attribution (coming soon). |
| Daily table size and maintenance cost (coming soon). |
| The jobs that run your notebooks, for notebook lineage (with Step 5). |
| The SQL that produces each table. |
Multiple workspaces and regions
Situation | What to do |
One OAuth connection covers your whole account | Foundational covers every workspace you added the service principal to, in every region. To narrow this, use the Included workspaces and Excluded workspaces options. The connector card names any workspace it skips and why. |
Each region needs a SQL warehouse | A metastore admin runs the Step 3 grants once per region. The service principal needs Can use on a warehouse in each region. Foundational picks a warehouse the principal can use in each region. If a region has none, Foundational skips its workspaces, and the connector card says so. |
Workspaces in another region | Don’t create a second connection for another region. |
Two connections on one metastore | Don’t create two connections on the same metastore. Both cover the same workspaces, and Foundational counts cost twice. |
More than one metastore | If your workspaces use more than one metastore, contact Foundational support so we know which metastore your dbt, Airflow and other connections use. |
Step 1: Create credentials
Choose OAuth or a personal access token. Only OAuth lets one connection cover more than one workspace. A personal access token covers only the workspace you create it in.
Option A (recommended): OAuth
Option A (recommended): OAuth
We recommend a dedicated service principal over a personal user.
In the account console, go to User management → Service principals. As a workspace admin, go to Settings → Identity and access → Service principals. Add a service principal, for example
foundational-connector.
Add it to every workspace you want Foundational to cover, with the Workspace access and Databricks SQL access entitlements.
Open the service principal and click Generate secret.
Copy the client ID and secret before you close the dialog. Databricks shows the secret only once.
Save the client ID and secret. You'll need them to connect Foundational to Databricks.
Continue with Step 2.
Option B: Personal access token
Option B: Personal access token
Create a personal access token for the service principal or for a user. We recommend a service principal. To create a token for a service principal, a workspace admin uses the Databricks token-management API (an on-behalf-of token), or signs in to the Databricks CLI as the service principal. Give it the maximum lifetime. If it expires, ingestion stops.
If your workspace uses scoped tokens, include the scopes
sql,unity-catalog,workspace,dashboardsandgenie. Addserving-endpointsonly if you use Foundational IQ on Databricks.
Copy the token before you close the dialog. Databricks shows it only once.
Save the token. You'll need it to connect Foundational to Databricks.
Continue with Step 2.
Step 2: Give Foundational a SQL warehouse
Note: We recommend a dedicated warehouse, so Foundational’s cost stays separate and small (a few cents a day). Create a serverless SQL warehouse with these settings: size 2X-Small, scaling max 1 cluster, auto stop 1 minute. If the UI does not offer 1 minute, set auto_stop_mins: 1 with the Databricks CLI or API.
On the SQL warehouse, open Permissions and give the service principal (or the user who owns the token) Can use. An existing warehouse is fine.
Copy the warehouse ID. In Connection details, it is the last part of the HTTP path, after
/warehouses/.
If your workspaces are in more than one region, repeat this step in a workspace of each region.
Step 3: Grant access (metastore admin)
Only a metastore admin can make system-table grants. A workspace admin cannot.
In the SQL editor, run the following once per region (metastore). Replace <principal> with the service principal’s application ID (or the user’s email).
-- System tables: query history, lineage and access audit, billing, compute, storage, jobs and pipelines
GRANT USE CATALOG ON CATALOG system TO `<principal>`;
GRANT USE SCHEMA, SELECT ON SCHEMA system.query TO `<principal>`;
GRANT USE SCHEMA, SELECT ON SCHEMA system.access TO `<principal>`;
GRANT USE SCHEMA, SELECT ON SCHEMA system.billing TO `<principal>`;
GRANT USE SCHEMA, SELECT ON SCHEMA system.compute TO `<principal>`;
GRANT USE SCHEMA, SELECT ON SCHEMA system.storage TO `<principal>`;
GRANT USE SCHEMA, SELECT ON SCHEMA system.lakeflow TO `<principal>`;
-- Every catalog you want Foundational to cover (repeat per catalog, including foreign catalogs)
GRANT USE CATALOG, USE SCHEMA, SELECT ON CATALOG <catalog> TO `<principal>`;
Why SELECT on catalogs: Foundational reads each table’s commit history, size and row count from table metadata. BROWSE alone shows names and columns only.
If you bound a catalog to specific workspaces, also bind the workspace you connect (Read only is enough). If you can't, contact Foundational support.
Step 4 (recommended): Show query text (account admin)
Databricks hides query text in system.query.history from anyone who is not an account admin. To let Foundational show the SQL that produces each table:
In the account console, open User management → Groups and create (or open) an account-level group named exactly
databricks_pii_access. A workspace-local group with the same name does not work.
Add the service principal (or the user who owns the token) to it.
Membership lets the principal read the text of all queries in the metastore.
Note: If you skip this step, everything else works, but Foundational cannot show the SQL that produces each table.
Step 5 (optional): Turn on notebook lineage
On each workspace folder that holds notebooks, open Share and give the service principal (or the user who owns the token) Can View. For notebooks that jobs run, also give Can View on the jobs.
When you connect, turn on Notebook lineage under Scan options on the Set Up New Connection screen. You can also turn it on later when you edit the connection. Without it, Foundational does not read notebooks.
Connect Foundational to Databricks
Select the option you chose in Step 1.
Option A (recommended): OAuth
Option A (recommended): OAuth
In Foundational, open the Integrations page.
In Warehouses, select the Databricks card and click Connect.
Click Add Workspace.
The initial setup screen opens. Click Start Setup.
The Set Up New Connection screen opens.
Enter the details:
Workspace URL: The top-level URL from your browser’s address bar.
OAuth Client ID: The client ID from step 1.
OAuth Client Secret: The secret from step 1.
Warehouse ID: The ID from step 2.
Foundational IQ — Databricks serving endpoint (optional): Enter it only if you use Foundational IQ on Databricks.
Scan options (optional): To add notebook lineage, turn on Notebook lineage. See Step 5.
Click Next.
The final setup screen opens. To complete the connection, click Save.
Option B: Personal access token
Option B: Personal access token
In Foundational, open the Integrations page.
In Warehouses, select the Databricks card and click Connect.
Click Add Workspace.
The initial setup screen opens. Click Start Setup.
The Set Up New Connection screen opens.
Enter the details:
Workspace URL: The top-level URL from your browser’s address bar.
Personal Access Token: The token from step 1.
Warehouse ID: The ID from step 2.
Foundational IQ — Databricks serving endpoint (optional): Enter it only if you use Foundational IQ on Databricks.
Scan options (optional): To add notebook lineage, turn on Notebook lineage. See Step 5.
Click Next.
The final setup screen opens. To complete the connection, click Save.
When you save, Foundational checks every grant above. The connection saves unless Foundational cannot list your catalogs. Any other missing grant shows as a warning on the connector card along with the SQL to fix it. The features that need that grant stay off until you grant it.
After you add the grant, click Re-check on the card. A card with no warnings means every grant is in place.
That's it. Foundational is now connected to Databricks.
Connector card warnings
If a grant is missing or Databricks refuses a feature, the connector card shows a warning.
Warning | What it means and how to fix it |
Warehouse access missing | Give the service principal Can use on the warehouse (step 2). |
System schema grants missing | A metastore admin must run the system grants in step 3. |
Catalog grants missing | The warning lists each catalog the principal can only browse, with the |
Query text hidden (information) | Add the service principal to |
Feature limited (table statistics, tags or Unity Catalog lineage) | Databricks refused one feature, and the message shows its answer. The rest of the scan, including lineage and schema, runs as usual, and Foundational keeps the previous values. |
Workspaces skipped (OAuth) | The warning names each workspace the connection doesn’t cover and why. Usually, you haven’t added the service principal to that workspace, or its region has no SQL warehouse the principal can use. |
Single workspace (personal access token) | The connection covers only its own workspace. To cover the rest of your account, see Switch an existing connection to OAuth. |
Switch an existing connection to OAuth
Existing connections that use a personal access token keep working. To switch:
Complete Option A (OAuth) in Step 1, then do steps 2 to 4 for the service principal.
Edit the connection and replace the token with the OAuth Client ID and OAuth Client Secret.
Click Save.
Lineage and history carry over.
Useful links
These links go to the AWS docs. Azure and GCP have equivalent pages.
