Foundational continuously monitors all your pending Pull Requests to ensure that any changes impacting your data stack—whether made in data engineering repositories or production code—do not lead to data incidents, pipeline disruptions, or compromises in data quality. This article delves into the benefits of this feature, how it operates, and how you can effortlessly configure it.
What makes Foundational CI essential?
In most organizations, the data stack is fragmented across multiple teams and repositories. For example, you might have an upstream production Postgres database managed through an ORM, ingested into Snowflake via Fivetran, transformed with dbt, and finally visualized in Tableau. In such a fragmented environment, a seemingly minor change to an upstream Postgres could inadvertently break a Tableau dashboard without the team’s knowledge. Foundational CI, powered by comprehensive end-to-end lineage, thoroughly checks every pending Pull Request to ensure it doesn’t disrupt downstream transformations, dashboards, ML models, and more. Moreover, these CI tests are seamlessly integrated into GitHub or GitLab, making them accessible to both the author and reviewer, regardless of their familiarity with Foundational.
How does it work?
After configuring a repository in Foundational (refer to: How To Setup GitHub), every pending Pull Request undergoes a thorough scan by Foundational. This scan determines downstream impact and checks for any potential issues or problems that might be introduced by the changes. The checks include:
Breaking schema changes
For instance, does the change involve removing or renaming a column that is still in use downstream (e.g., in a Tableau dashboard)?Semantic bugs
For more information, see our blog post on semantic bugs.
The check results are directly integrated into the Pull Request checks (rather than just appearing as comments), ensuring they remain visible to both the author and reviewer. By clicking on the check results, users can see which specific check failed and access more detailed information. Additionally, users have the option to click through to the Foundational Pull Request page to delve deeper into the Pull Request details or to manually mark these tests as skipped.
Failed checks, as they appear in GitHub
If all checks are successful and Foundational confirms that the Pull Request will not result in any issues or data quality incidents, the Pull Request checks are marked as passed. This assurance allows the author and reviewers to confidently proceed with merging and deploying the Pull Request
Successfull checks, as they appear in GitHub
Making Checks Mandatory for Pull Request Merges
By default, the checks are non-blocking, allowing developers to merge pending Pull Requests even if the Foundational checks fail. You can, however, configure these checks to block Pull Request merges if they fail, ensuring that any Pull Request flagged by Foundational as problematic cannot be merged.
To do this, see: Configure "Foundational Data Issue Analysis" to Block PR Merges