How Do You Test Systems That Analyze Behavior Over Time?

We borrowed a technique from quantitative finance to test systems that process patterns over time, and it caught bugs no unit test ever would.

Most software tests ask a simple question: given this input, do I get the right output? Backtesting asks something harder: given a realistic stream of behavior over time, does the system still get it right?

If you build analytics pipelines, anomaly detection, usage-based billing, or recommendation engines, you’ve felt this gap. Unit tests pass. Integration tests pass. Yet real data breaks everything because it has a shape those tests never covered.

We hit this wall and found our answer in an unlikely place: quant finance.

Key Takeaways

• Poor software quality costs the US $2.41 trillion per year (CISQ, 2022). Much of it comes from bugs that only surface under realistic conditions.
• Synthetic scenario generators (borrowed from quant backtesting) catch temporal bugs that unit and integration tests miss
• Golden baselines provide regression safety nets for systems that produce continuous scores, not binary pass/fail
• Start with one scenario targeting your most common user pattern; add realism over time
• An AI agent can drive the whole loop cheaply if the harness exposes async run/poll endpoints, a single JSON summary per suite run, and a durable debug artifact per scenario

Why Do Normal Tests Miss Temporal Bugs?

Two-thirds of technology projects end in partial or total failure, based on the Standish Group’s CHAOS database (CHAOS Report, 2020). For applications that process time-series inputs, the problem is worse: bugs hide in the shape of the signal, not the values.

Consider a system that processes a stream of user activity over days and produces a score or a decision. Its logic depends on:

• Density: how often events arrive (bursts vs. steady vs. sparse)
• Repetition: real activity has noise: repeated actions, idle re-reads, duplicate signals
• Gaps: periods of inactivity your system must interpret (session boundaries, churn signals)
• Saturation: derived metrics that plateau after a threshold
• Progression: meaningful change that builds up over time

You can unit test each of these in isolation. In practice, though, bugs live in how stages interact when fed inputs with realistic structure. A test with 3 events spaced exactly 60 seconds apart tells you nothing about a real user’s erratic Tuesday afternoon.

What Does Quantitative Finance Already Know?

Quant trading firms don’t test strategies on a single price point. They build fake market histories (datasets with known traits like trends, crash events, and calm periods) and check that strategies respond correctly to each one.

Here’s the key: they control the properties of the input and assert on the pipeline’s result. Not “does this function do math right?” That’s a unit test. The real question: “given a market that behaves like this, does our system make the right call?”

A NIST study found that software bugs cost the US economy $59.5 billion per year, with defect correction eating roughly 80% of total development costs (NIST Planning Report 02-3, 2002). Most of that waste traces back to bugs that only show up under realistic data conditions. That’s exactly what backtesting catches.

This is backtesting. And it works for product engineering too.

Unit Tests vs. Backtesting

Here’s how they compare:

	Unit Tests	Backtesting
Input	Crafted individual values	Synthetic temporal sequences
Scope	Single function or module	Full pipeline, all stages
What it checks	”Does this function compute correctly?"	"Does the system behave correctly given realistic patterns?”
Bug surface	Logic errors in isolated code	Interaction effects across stages, noise sensitivity, saturation
Maintenance	Update when code changes	Update when production reveals new patterns
Best for	Stateless request/response	Temporal streams, scoring, analytics

To be clear, neither replaces the other. Backtesting fills the gap where unit tests are structurally blind.

How Does the Technique Work?

The core loop has five steps.

1. Catalog What Your System Is Sensitive To

Before crafting any inputs, list the time-based behaviors that affect your results:

• An alerting system is sensitive to spike shape, baseline noise, and gap duration
• A billing pipeline is sensitive to burst patterns, timezone boundaries, and plan tier thresholds
• A recommendation engine is sensitive to signal consistency, preference drift, and session context

These are your test dimensions. Every generated test case should target a specific combination.

2. Build Generators, Not Fixtures

Static JSON fixtures go stale and nobody remembers why they look the way they do. Write generators instead: small programs that produce event streams with clear, documented properties.

Generator("steady-growth") →
  60 days of activity
  Events every 10-30 seconds during sessions
  Gradual content progression
  30% duplicate/noise events
  2-hour work blocks with 30-minute gaps

In practice, the code is straightforward. Here’s what that spec looks like as a real generator:

function generateSteadyGrowth(seed = 42, days = 60, noiseRatio = 0.3) {
  const rng = createSeededRng(seed); // same seed = same output, every time
  const events: { timestamp: string; action: string; progress: number }[] = [];
  const start = new Date("2025-01-01T09:00:00");

  for (let day = 0; day < days; day++) {
    for (let block = 0; block < 3; block++) { // three 2-hour work blocks
      let t = addHours(addDays(start, day), block * 2.5);
      const end = addHours(t, 2);
      while (t < end) {
        const event = {
          timestamp: t.toISOString(),
          action: rng.pick(["view", "edit", "navigate"]),
          progress: day / days, // gradual progression
        };
        events.push(event);
        if (rng.float() < noiseRatio) events.push({ ...event }); // 30% duplicates
        t = addSeconds(t, rng.int(10, 30));
      }
    }
  }
  return events; // ~150k events, fully deterministic
}

The seeded RNG is the key detail. Every property from the spec (density, noise ratio, progression curve, session gaps) maps to a parameter you can tune. When a check fails, you read the generator and understand what behavior it was targeting. Good luck doing that with a 2,000-line JSON file.

Generators should be repeatable. In other words, same seed, same result, every time. Failures stay reproducible and baselines stay stable.

3. Run the Full Pipeline

This isn’t a unit test. Feed the generated inputs into your actual production code path. If your application has ingestion, analysis, and scoring layers, run all three. The bug you’re hunting probably lives in the handoff between the second and third layer when the input has a property the second layer doesn’t preserve.

Establishing and Iterating Baselines

4. Record Golden Baselines

For each test case, record the expected result:

steady-growth scenario:
  Score: 85.5%
  Detected sessions: 60
  Active time: 127 hours
  Progression: 0.92

Our finding: Golden baselines caught more regressions than any other testing investment we’ve made, especially when refactoring analysis layers that produce continuous scores rather than binary pass/fail results.

As a rule, store baselines alongside generators. Review them like code. When a baseline changes, the commit message should explain why the output shifted, not just update the number.

5. Iterate on Realism

Your first generated inputs will be too clean. As you find production issues, ask: “Would any of our simulations have caught this?” If not, add a new one. Over time, your test library becomes a catalog of every behavior your system needs to handle correctly.

Can You Drive This Loop With an AI Agent?

Once the pieces exist (generators, a runner, golden baselines, regression thresholds), the loop becomes something an AI agent can execute end-to-end: propose a change, rerun the suite, read the regression summary, keep or revert. A human just reviews the diff. The deeper argument for this mode of working, treating AI as a team member rather than a chat window, is what makes it worth the setup cost.

What makes this practical is the shape of the harness, not the cleverness of the agent. A poorly shaped harness will burn tokens on orchestration. A well-shaped one lets the agent spend its budget on the actual scoring questions.

Here are the design choices that matter.

Run the harness inside the real app, not a separate CLI

The most expensive move an agent can make is bootstrapping the system under test from scratch every iteration. If your app has encrypted local state, account auth, or a long-lived background process, you do not want the agent reinventing that path. Expose the harness as an HTTP endpoint inside the real application. The human launches the app and unlocks whatever needs unlocking. The agent then drives the already-running process through a small localhost API. Pair this with local code intelligence if the agent also needs to reason about the source it is tuning, rather than just the outputs.

That single decision removes a class of brittle setup from every run.

Async run-and-poll, not blocking exec

Keep the API surface tiny:

POST /harness/run       -> { job_id, status: "queued" }
GET  /harness/jobs/:id  -> { status, result?, artifact_path? }

Four states (queued, running, completed, failed) cover everything. An agent that can poll does not have to babysit a seven-minute run, does not have to stream log output, and does not lose context if a job is slow. It just checks back.

Emit one structured summary, not raw logs

This is the token-cost fix.

Give the agent a single script that runs the whole suite, compares every result against its baseline, applies your regression thresholds, and prints one JSON blob: per-scenario score, delta, regression: true | false. The agent reads that. Nothing else.

Only when a regression flag fires does it open the relevant debug artifact and dig deeper. Most runs stop at the summary. The difference between “agent tails stdout across fourteen scenario runs” and “agent reads one small JSON” is roughly the difference between a session that costs a few dollars and one that costs tens. If you do not already track how your AI coding sessions spend tokens, this is the point at which you will want to.

Encode regression thresholds in the runner, not the prompt

If your policy is “golden paths may not drop more than 2%, negative controls may not rise more than 5%”, put that logic in the script. The JSON arrives with regression: true | false already decided. The agent does not have to hold the rules in context, and you do not have to trust it to apply them consistently across iterations.

Give every run a durable artifact

Each completed run should drop a debug dump to disk: inputs, intermediate state, final scores, enough metadata to diagnose a regression without rerunning. Rerunning is expensive. Opening a file is nearly free. A durable artifact per run is one of the highest-leverage choices you can make for iteration cost.

A short runbook for the agent

Write down a few operating rules. They stop wasted cycles more reliably than any amount of prompting.

• One change per iteration, so regressions are attributable.
• After a backend code change, restart the app before trusting results.
• Do not start a new run while one is already queued or running.
• If a result surprises you, rerun the same inputs once before editing anything.
• Treat any score/baseline mismatch as real until proven otherwise.

None of this makes the agent smarter. It makes the loop cheap, deterministic, and mostly unattended, so the agent spends its budget on the scoring and matcher questions you actually care about.

What Does Backtesting Catch That Unit Tests Don’t?

Two-thirds of technology projects end in partial or total failure, according to the Standish Group’s CHAOS database (Standish Group CHAOS Report, 2020). Many of those failures trace to integration bugs that no single test caught. Here’s where backtesting pays off:

Saturation bugs. A derived metric works fine for the first 100 events, then plateaus because of a windowing assumption. Your unit test with 5 events never hits the ceiling.

Session boundary edge cases. Your system infers sessions from activity gaps. A 29-minute gap is one session; a 31-minute gap is two. That difference cascades through every per-session metric. No unit test covers this because no unit test has realistic gaps.

Noise sensitivity. Real users generate duplicates: idle re-reads, repeated clicks, background refreshes. A simulated replay with 30% duplicates surfaces double-counting right away.

Interaction effects. Your dedup layer removes 40% of events. Your analysis layer computes density. Your scoring layer penalizes low density. Each layer is correct on its own. Together, they penalize users whose activity happens to be repetitive. Only an end-to-end test with realistic repetition catches this.

Practical Advice for Getting Started

Start with one test case. Pick the most common user behavior. Build a generator, run the application, record the reference result. Edge cases come later.

From there, keep generators in-repo. They’re test infrastructure, not throwaway scripts. Version them. Review changes. They’re as important as the code they test.

Also, separate generation from execution. One program produces the data file. A separate runner loads it into your application. This way, you can regenerate inputs without re-running checks, and vice versa.

Once that’s stable, automate the loop. The highest-value version is a CI step that generates test cases, seeds your application, runs the processing chain, and compares against reference results. Manual replay testing helps during development; automated runs stop regressions.

Finally, name test cases by behavior. steady-growth-60-days tells future-you what it exercises. test-data-v3-final tells you nothing.

When Should You Use This Technique?

This approach earns its keep when:

• Your system processes temporal sequences, not individual requests
• Correctness depends on statistical properties of the input (density, distribution, gaps)
• Bugs tend to appear at stage boundaries in your pipeline, not inside any single function
• Real-world data has noise patterns that clean test data can’t capture
• You need regression detection for continuous scores rather than binary pass/fail

A few examples where we’ve seen this work: activity scoring engines that aggregate weeks of user behavior, billing pipelines that compute usage across timezone boundaries, and anomaly detection systems that need to distinguish a real spike from normal noise. In each case, the bug surface was in the interaction between stages under realistic conditions, not in the logic of any individual component.

If your system is a stateless request/response API, this is overkill. Unit and integration tests have you covered. But if your system looks at a stream of activity over time and draws conclusions, start backtesting your own product. The same direction of test extends past application code: AI agent sessions can be recorded, replayed, and judged with the same discipline, and I cover that in Backtesting AI Agents.

We use this approach to test a system that analyzes user activity over days and weeks, producing confidence scores from time-based signals. The generated test cases surface bugs that no unit test would catch, especially around signal saturation, session boundaries, and noise tolerance. Over six months, we grew from one simulation to twelve, and the reference outputs have blocked more regressions than any other testing investment we’ve made.

Frequently Asked Questions

What’s the difference between backtesting and property-based testing?

Property-based testing throws random inputs at a single function to check its invariants. Backtesting generates realistic sequences over time to check how your whole system behaves across stages. The bugs that escape to production tend to live in the handoffs between layers, not inside any single function. That’s the gap backtesting fills.

How many synthetic scenarios do I need to start?

One. Start with a single generator that represents your most common user pattern. Record the golden baseline and add it to CI. Add edge-case scenarios as production issues reveal gaps. We started with one “steady-growth” scenario and grew to twelve over six months.

Can an AI agent run this whole loop for me?

Yes, once the harness is shaped for it. Three design choices make it cheap: an async run/poll API (so the agent never babysits a long job), a single runner that compares every scenario against its baseline and prints one JSON summary (so the agent reads one short result, not fourteen raw logs), and a per-run debug artifact on disk (so investigations open a file instead of rerunning). Encode your regression thresholds in the runner itself, not the prompt, so the agent does not have to reason about them on every iteration.

Do golden baselines break every time I change the code?

They break when output changes, which is the point. If you refactor the analysis layer and the score shifts from 85.5% to 84.2%, you’ve caught a regression. The commit message should explain why the shift is acceptable, or the test should block the merge. With two-thirds of tech projects ending in partial or total failure (Standish Group CHAOS Report, 2020), this kind of regression visibility is rare and worth the setup cost.