Sort your data checks by what happens when they fail

The usual framing is "data observability vs data quality testing", as if a team picks one product category and the other becomes optional. That framing hides the more useful question. For every check on a dataset, what happens when it fails? Does a candidate table stay hidden, does someone get paged, or does a number go on a dashboard nobody opens?

Once you ask that, the tool labels matter a lot less.

Tests are for things you already know

A data test encodes a rule someone can write down in advance. Order IDs are unique. Every customer_id in orders exists in customers. Status is one of five values. The ledger total matches the source total.

dbt describes its data tests in exactly these terms: a test is a select that returns the failing records, and it passes when it returns zero rows. Assert uniqueness and the query looks for duplicates. Severity is configurable, so the same assertion can warn or error, with warn_if and error_if thresholds on the count of failing rows.

That severity setting is where the real design work is. A blocking test with a high false-positive rate gets bypassed within a few weeks, usually by someone who is right to bypass it. A test that only warns, with no owner and no response time, is telemetry wearing a test's name.

Two practical details that are easy to miss:

• If a threshold recalculates itself from recent history, it is no longer a deterministic test. Same data, different day, different verdict. It behaves like a monitor and needs to be governed like one.
• Stored failing rows are often the most sensitive rows in the table. Cap the sample, restrict who can read it, and keep it out of a widely readable bucket.

Monitors are for things nobody predicted

Observability covers the failures nobody wrote a rule for: a source that silently stopped sending one region, a partition that arrived half full, a distribution that shifted because a product launched. Freshness, volume, schema, distribution and lineage signals catch these because they compare the system against its own history rather than against a written obligation.

The catch is that legitimate change looks exactly like failure. Backfills, seasonality, launches and a changing product mix all move volume and distribution. So it depends on the signal, but most anomaly alerts should start as warnings with human review, and only get promoted to blocking after someone has measured their false-positive rate and defined a safe override.

Signals are only useful for diagnosis when they share identity. OpenLineage's Data Quality Metrics facet is a decent model here: rowCount, bytes, fileCount, lastUpdated, a captureDate, and per-column nullCount, distinctCount, min, max and quantiles, all attached to a dataset inside a run event. Metrics tied to a run and a dataset can be joined to the upstream job that produced them. Metrics on a standalone dashboard force whoever is on call to rebuild that join by hand during the incident.

The same check can play two roles

A row-count assertion run after a batch lands is detection: the batch already happened. If that same assertion gates whether the candidate table is published, it is also prevention. A freshness monitor can tell you an upstream is three hours late and can do nothing to make the data arrive.

So classify by decision, not by product. For each check, write down six things: the asset, the condition, when it is evaluated, what it does on failure, who owns it, and where the evidence goes. If you cannot fill in the consequence column, the check is profiling. Profiling is fine, just keep it separate from paging.

Coverage comes from failure modes

Three tools that all alert on row count are one signal paid for three times. A better coverage map starts from what can actually go wrong with a specific data product: missing source, partial partition, duplicate replay, stale reference data, schema break, late events, a delete that did not propagate. Map each one to the control that prevents it, the one that detects it, and who fixes it. The empty cells are the real backlog.

Priority should follow decision impact rather than query volume. A small regulatory export can justify stricter reconciliation than the most popular exploratory table in the warehouse.

The useful loop is the one that feeds back. When a monitor catches a failure twice, that failure is no longer unknown, and it should become an explicit test or an engineering fix upstream.

The longer data observability vs data quality testing guide on Edilec has the full control-role table, a signal-by-signal comparison of tests and monitors, and a seeded-failure plan for evaluating tools before buying one.

Disclosure: AI assistance was used to draft this excerpt. The linked Edilec article, the dbt data tests documentation and the OpenLineage Data Quality Metrics facet specification are the public technical sources; this excerpt does not describe a client deployment or measured Edilec result.