Frameworks

Data Observability vs. Data Testing: When Each One Actually Works

Q: Can you replace dbt tests with data observability?

No. Observability cannot know that `order_id` must join uniquely to `payment_id`, or that `country_code` must be in ISO 3166. Those are business rules only humans can encode. Observability catches drift; testing catches rule violations. A mature stack runs both, on different tables, at different times.

Aug 21, 2026 · 5 min read · The Dalanio team

Every data team eventually asks the same question. We have dbt tests. Do we still need observability? Or the reverse. We are shopping observability tools. Can we skip the tests?

Both framings are wrong. Testing and observability catch different classes of failure. Choosing between them is like choosing between locks and burglar alarms. This is the framework that decides where each one belongs.

What does data testing actually do?

Testing asserts a rule you wrote. not_null on a primary key. unique on an ID column. accepted_values in ('paid', 'refunded', 'pending'). relationships between a foreign key and its parent. Whatever you can express as a boolean SQL check runs on every model build and fails loudly.

Testing is strongest on three axes.

Explicit business logic. Only you know that a valid ARR row has arr > 0 and mrr = arr / 12 within a rounding tolerance. No tool can infer that from data.
Post-transform assertions. After your dbt model has computed active_customers, testing confirms the number matches your definition. Observability cannot check semantic correctness.
Deterministic pass or fail. Either the row passes or it does not. No baselines, no windows, no probability.

The failure mode of testing is coverage. Nobody writes a test for the thing they did not think of.

What does data observability actually do?

Observability learns what a table normally looks like and flags deviation. Four dimensions matter.

Dimension	What observability watches	What it catches
Freshness	Load cadence and lag	Ingestion stopped or slowed
Volume	Row counts vs. seasonal baseline	Data drop, backfill spike
Schema	Column names, types, nullability	Renames, drops, retypes
Distribution	Nulls, cardinality, percentiles	Silent corruption, semantic drift

None of those require a rule. You do not tell the tool "expect 1.2M rows on Tuesdays." It watches your history, learns the pattern, and alerts on the deviation from it.

The failure mode of observability is context. It can tell you that something is off. It cannot tell you whether "off" matters to the business.

Where do the two tools belong in a pipeline?

Split your warehouse into three layers, then apply the rule.

Raw / landing. Tables you did not write, loaded from Fivetran, Segment, Airbyte, or app databases. Observability dominates here. You cannot write dbt tests on a schema you do not control.
Staging / intermediate. Tables you own, still close to the source. Both testing and observability apply. Basic tests on primary keys and joins; observability on volume and freshness.
Marts / models. Tables the business consumes. Testing dominates here. This is where your business logic lives, and this is where wrong numbers do the most damage.

The mistake most teams make is inverting this: heavy testing on marts, nothing on raw. When the source table breaks, no test on the mart can save it because the model still runs, just on stale or shifted data.

When should you reach for data contracts?

Contracts move the check upstream to the producer. A contract on raw.stripe.invoices says: this schema, these columns, these types, this loading frequency, or the producer's CI fails and the change never lands.

Contracts are the right tool in three cases.

Recurring producer-side breakage. If the same team keeps renaming columns in your Postgres replica, monitoring is treating a symptom. A contract at the ingestion layer stops the incident from opening at all.
Cross-team dependencies. When the producer is a different engineering team, formalize the interface. This is how APIs work; there is no reason data pipelines should not.
Regulated or SLA-bound data. If a table feeds a report that goes to auditors or customers, you cannot afford to catch drift after the fact.

Contracts are the highest cost of the three approaches, because they require producer buy-in. They are also the only one that prevents breakage rather than detecting it.

How do you decide the mix for a specific table?

A short rubric works. Score the table on five questions, then apply the mapping.

Do I own the schema? (Yes / No)
Is the business logic complex enough to encode as rules? (Yes / No)
Does the load cadence follow a learnable pattern? (Yes / No)
Is the producer a different team or a vendor? (Yes / No)
Does the table feed a customer-facing or regulated report? (Yes / No)

Then:

Own the schema, encodable rules, learnable cadence: tests + observability.
Do not own the schema, learnable cadence: observability only.
Different-team producer, recurring breakage, regulated report: add a contract on top.
Ad hoc analytics table nobody reports off of: skip both, deprecate the table.

The last one matters more than the others. The cheapest data quality investment is deleting tables nobody depends on.

What is the cost tradeoff between the three?

Rough order of magnitude, per 100 tables per year:

Tests only. $0 in tool cost. 40 to 80 engineer hours to write, plus 4 to 8 hours per month maintaining. Coverage on business logic, blind on freshness, volume, and schema.
Observability only. $12K to $60K per year in tool cost depending on vendor and table count. 4 to 8 hours to onboard. Coverage on freshness, volume, schema, distribution. Blind on business logic.
Both. Adds the tests cost on top of observability. Combined coverage is the only version that catches both known and unknown failures.
Contracts on top. 20 to 60 engineer hours per producer team to negotiate and enforce. Highest cost, catches issues before they enter the warehouse.

Skipping observability to save the tool cost is the most common mistake. The finance impact of one uncaught silent incident (see the "hidden cost" analysis) outweighs a full year of observability spend for most teams above 200 tables.

How do you know your current mix is wrong?

Three tells.

Your CEO or another executive is regularly the first to notice a broken number. Coverage on the marts layer is fine; you are blind on freshness upstream.
Your dbt run passes but the dashboards show wrong data. Business logic is validated; the input to the logic drifted and nothing caught it.
Same producer breaks something monthly. You are monitoring the effect; you need a contract at the cause.

Any one of these tells means the mix is off. Fix the mix before adding more of what you already have.

The mistake to avoid

Data teams treat observability and testing as competing purchases. They are not. Testing is your business-logic guardrail on the marts you own. Observability is your smoke detector across every raw and staging table where you cannot write rules fast enough. Contracts are the fire code that prevents fires upstream. Buy the layer that matches where your incidents actually originate, not the layer that is easiest to deploy. The team that runs all three on the right tables catches breakage that never reaches a dashboard.

data observabilitydbt testsdata contractsdata reliability

Frequently asked questions

Can you replace dbt tests with data observability?

No. Observability cannot know that order_id must join uniquely to payment_id, or that country_code must be in ISO 3166. Those are business rules only humans can encode. Observability catches drift; testing catches rule violations. A mature stack runs both, on different tables, at different times.

When should you use data contracts instead?

When the same schema change breaks your pipeline more than twice, or when the producer of a table is a different team (engineering, a vendor, a partner). Contracts push validation upstream to the producer, so a breaking change fails at their CI, not your morning dbt run. Contracts do not replace monitoring, they reduce its load.

How much of my table coverage should be tests vs. observability?

Roughly 20% tests, 80% observability, weighted by table count. Tests concentrate in the model layer where you know the business logic. Observability covers raw and staging where you do not. Inverting this ratio is the most common failure mode: teams write 500 dbt tests and still get paged by their CEO because nothing was watching the source table's load cadence.

Does data observability work without lineage?

Barely. Without lineage, you get anomalies but no blast radius, so every alert requires human investigation to answer 'what breaks if this is wrong?' Column-level lineage from query logs and dbt manifests is what makes observability actionable instead of noisy. Skip lineage and you are running a fancy null-count monitor.

Is data observability just monitoring rebranded?

Partly. The rebrand added three things monitoring did not have: learned baselines instead of thresholds, root-cause grouping via lineage, and column-level anomaly detection. If a tool sells 'observability' but only offers threshold alerts on row_count, it is monitoring with a new label. The learned-baseline and lineage pieces are the real difference.

Catch broken data before dashboards do

Dalanio learns each table's normal on Snowflake, BigQuery, Redshift, and dbt, then pages your team when freshness, volume, or schema drift.

Request early access