Back to blog

Engineering

Regression Detection Without Explicit Tests

8 min read
Abstract visualization of code dependency paths

The standard answer to regression prevention is "write more tests." The advice is correct as far as it goes. A comprehensive test suite is the most reliable defense against regressions in paths the test suite covers. The problem is the qualifier: the test suite covers what it covers, and regressions that happen in untested paths have no automatic defense.

In practice, most codebases have significant untested surface area. Some of it is legacy code that predates the team's testing culture. Some of it is code that is genuinely hard to test -- code with external dependencies, timing-sensitive behavior, or side effects that test infrastructure can't easily simulate. And some of it is code that has tests, but tests that verify the happy path and not the edge cases where regressions actually tend to appear.

The question is: what can you do about regressions in code paths you don't have tests for?

Understanding Where Regressions Come From

A regression is a change that breaks behavior that was previously working. Most regressions are not the result of directly modifying broken code. They are the result of modifying code that has implicit dependencies on behavior that another piece of code relied on.

The developer making the change was focused on the explicit behavior described in the ticket. The regression appeared because a downstream consumer of the modified function had an implicit assumption about its behavior that the developer didn't know existed and that no test documented.

This is why regressions are hard to prevent purely with tests. You can write a test for the behavior you know about. You can't write a test for the implicit dependency you don't know exists until it breaks.

Static Dependency Analysis

One approach to regression detection without explicit tests is static call graph analysis: trace which functions call the modified function, and flag the ones that might be affected by the change.

This is useful but has limits. Static analysis can tell you that processPayment calls validateAmount, and that validateAmount was modified. It can't tell you whether the modification to validateAmount changed the behavior in a way that breaks processPayment's expectations. That evaluation requires understanding what validateAmount is expected to do and what the PR actually changed about it.

Call graph analysis is good at surfacing the affected surface area. It is not good at evaluating the risk at each affected call site.

Change Semantic Analysis

The more powerful approach is to analyze what the PR actually changed about the modified function's semantics and then evaluate whether the callers handle those changes correctly.

For example: if a PR changes a function from always returning a non-null value to sometimes returning null (to handle a new case), the semantic change is "this function is now nullable." Callers that dereference the return value without a null guard are now potentially broken, even if they weren't before. Identifying those callers and flagging them is regression detection without requiring tests to already exist.

This kind of analysis requires understanding both the original behavior and the new behavior at the semantic level -- not just which lines changed, but what the change means for how the function behaves. This is where automated code review that combines static analysis with semantic understanding adds value beyond what pure call graph tracing provides.

Test Coverage Gap Identification

A complementary approach is not to detect the regression directly, but to detect the absence of test coverage for the changed code path and flag it as a risk before merge.

When a PR modifies a function that handles a specific edge case -- say, the case where a user's account is in a pending-deletion state -- the review can check whether any test in the PR exercises that path. If the edge case is being added or modified but no test covers it, the review flags the coverage gap with a specific description: "the pending-deletion code path in getUserStatus has no test coverage; regressions in this path won't be caught by CI."

This doesn't prevent the regression if the code is actually wrong. It surfaces the risk clearly enough that the developer can decide whether to add coverage before merging or to document the gap explicitly. That is a better outcome than merging under the assumption that CI coverage is complete.

Historical Pattern Matching

A final approach is to flag code patterns that have historically been associated with regressions in the repository. If a specific pattern of async operation handling has produced three incidents over the past year, and a new PR contains the same pattern applied to a new code path, that is worth flagging even if the new code is technically correct -- because the same pattern in slightly different contexts has failed before.

This kind of history-aware analysis isn't available from a tool that sees each PR in isolation. It requires accumulated knowledge of what has broken in this codebase specifically. For teams that have been using automated review long enough to build that history, it becomes one of the more valuable forms of regression prevention: not "this pattern is generically risky," but "this specific pattern has caused incidents in this codebase, here are the three examples."

None of these approaches replaces a comprehensive test suite. But for the significant portion of production bugs that originate in undertested code paths, they provide meaningful coverage of a surface area that CI cannot reach on its own.