How to Build a Data Quality Monitoring System That Actually Catches Breakage

Every data team has the same story. A dashboard the CFO looks at every Monday showed the wrong number for three weeks. The source table quietly stopped loading on a Tuesday, nobody noticed, and every downstream model happily served yesterday's stale rows as if they were fresh. The Slack thread started with your CEO, not your on-call.

A data quality monitoring system is supposed to prevent exactly that. Most of them do not, because they are collections of hand-written assertions rather than a system. This is the version that actually works.

What is a data quality monitoring system and why does it fail?

A data quality monitoring system is a set of automated checks that continuously watch every table your business depends on, detect deviation from normal, and route incidents to a named owner with downstream impact attached. In theory, it catches breakage before it reaches a dashboard. In practice, most systems break in three predictable ways.

• Coverage gaps. The system has dbt tests on the 60 models the analytics team owns. It has no coverage on the 400 raw and staging tables that feed those models, which is where 70% of breakage originates.
• Fixed thresholds. Somebody hardcoded row_count > 1000. The table's normal Monday volume is 200. Every Monday, an alert fires. Every Monday, it gets muted.
• Ownership drift. The alert routes to #data-alerts, which 40 people have muted. Nobody actually owns the response.

If any one of those is true, the system looks like monitoring but is not one.

What monitor types do you need to run in parallel?

Four, because that is the smallest set that catches the common failure modes. Skip any one and you leave a class of breakage undetected.

Freshness and volume catch about 65% of incidents. Schema catches roughly 20%. Distribution catches the remaining 15%, which are the ones that matter most because they are silent and often the reason a model quietly goes wrong for weeks.

How do you tier tables by criticality?

Not every table deserves an on-call page. Three tiers is enough.

• P0. Feeds a customer-facing feature, a finance number that ships to the board, or a regulated report. Pages on freshness, volume, schema, and distribution. Under 15% of your tables.
• P1. Feeds a production dashboard used weekly by a VP or head. Posts to the owning team's Slack. Roughly 30 to 40% of tables.
• P2. Everything else that is still governed. Daily digest, no interrupt. The remaining 45 to 55%.

If every table is P0, nothing is. If your incident feed cannot be scanned in 90 seconds by an on-call engineer, the tiers are wrong.

What baseline do the monitors actually learn?

A useful baseline is not a static rule. It is a rolling model of what each table normally does, with three properties.

1. Seasonality aware. Monday-through-Friday cadence, month-end batch jobs, and Black Friday spikes are learned, not alerted on.
2. Confidence banded. Anomaly is defined as sitting outside the 99th percentile of learned behavior for that hour of that day of week, not "20% deviation from mean."
3. Fast to bootstrap. Replay 4 to 8 weeks of warehouse history at connect time so day-one alerts are already useful, instead of waiting a month.

Hand-written rules land in one of two failure modes: too tight and alert-fatiguing, or too loose and missing real drops. Learned baselines let you skip the tuning treadmill entirely.

Who owns each incident?

Every incident has three roles. The system must resolve all three, on the row, at page time.

• Table owner. The engineer or analytics engineer whose team writes to the table. Owes an acknowledge within 15 minutes and a root cause within 4 hours for P0.
• Domain owner. The head or lead of the function that consumes it (revenue ops, finance, product). Owes the business context on whether to hold reports or ship a note to stakeholders.
• On-call. The rotating data engineer taking the interrupt this week. Owns escalation if the table owner is unreachable.

When the row cannot resolve a table owner, the default should be to fall through to the on-call rotation with a warning that ownership is missing. Untracked tables are the ones that break silently, then take 3 days to fix because nobody knows the schema.

How do you group incidents to prevent alert storms?

One source table breaks. Twelve downstream dbt models fail their tests. Four dashboards go stale. Without grouping, you get 17 pages for one root cause.

Group by three signals, in this order.

• Lineage. If model B depends on table A, and A has an open freshness incident, B's staleness is a downstream, not a new incident.
• Time. Anomalies opening within 90 seconds of each other on tables in the same schema are grouped.
• Cause. A schema change that breaks 8 queries in the same run is one incident with 8 impacted assets, not 8 incidents.

The measurable outcome is incidents per week per engineer under 5. Above that, the on-call is drowning and will start muting channels within 6 weeks.

What does a working weekly review look like?

Thirty minutes. One meeting. Same rhythm every week.

• Attendees. Head of data plus one analytics engineer and one data engineer. Domain owners join only when their table shows up.
• Agenda. All P0 incidents from the past 7 days, plus the top 5 P1s by downstream impact.
• Output. Every recurring incident either gets a fix committed within 2 weeks or gets promoted to a permanent monitor rule. Nothing lingers as "known noise" past two reviews.

Track five KPIs across the meeting: incidents opened, mean time to detect, mean time to acknowledge, mean time to resolve, and false-positive rate. If MTTD is above 60 minutes on P0, coverage is broken. If MTTA is above 30 minutes, ownership is broken.

The mistake to avoid

Most data teams treat monitoring as a project that ends when the first 100 dbt tests are green. That framing guarantees the system decays the day someone forgets to add a test to a new model. Treat monitoring as an operating layer instead: learned baselines on every governed table, four monitor types running in parallel, one owner per row, and a weekly review that turns recurring incidents into permanent fixes. Everything else is decoration on a dashboard that is still showing stale data.