Case Study
The System Worked. The Model Underneath It Didn’t.
How a reporting surface grew into a production operating system—and then exposed a flaw in the model underneath its own numbers.
The situation
I built the first version because I couldn’t understand what was happening.
I had stepped into the work while covering for someone on leave. The operating picture was spread across a CRM, spreadsheets, presentation decks, project boards, and other artifacts. Meetings meant moving between places and reconstructing the numbers before we could talk about them.
The problem was not that nothing worked. It was that the current state was hard to explain. More than one model described who belonged in the work, planning and capacity lived elsewhere, and list building was manual. No one could keep up with what was happening.
So the first move was modest: build something I could understand, then use it to make the meetings easier for everyone else too.
How it grew
Making the current state visible exposed the next question: could the team actually work what we were planning? That led to capacity planning.
Planning exposed a trust problem. A plan built on stale, inconsistent, or poorly defined data was not useful. Freshness, quality checks, consistent labels, and reviewable definitions became part of the product.
Once the populations were defined, the system needed to turn those decisions into correctly structured downstream work. Once the work was running, it needed close-out decisions, accountability, and next actions. Once people relied on the numbers, “why did this change?” became part of the product too.
Every capability existed because using the previous one exposed another operational need. The system grew from seeing, to planning, to trusting, to defining, to building, to running, and finally to investigating.
The operating system
Over roughly three months, the reporting surface became a daily production operating system representing more than 100,000 accounts.
It brought planning, population definitions, quality checks, work creation, execution decisions, reporting, investigation, and system health into one operating model. Operators and senior leaders could plan from the same surface, investigate discrepancies instead of merely presenting them, and route changes through the system.
I built it with AI as my development partner. I owned the operating model, product direction, requirements, definitions, domain judgment, validation, testing expectations, priorities, and architecture decisions. AI helped turn those decisions into working software much faster than I could have built it alone.
It was substantial. But the most useful thing it eventually showed me was not another number.
The turn
Production use exposed a flaw in the model underneath the numbers.
We were often treating membership in a wave of work as a changing query: who qualifies based on what is true right now? That sounds reasonable until the population also represents a historical business decision: who did we actually decide to work at that point in time?
Those are different things. Success could change an account enough that it disappeared from the population used to measure that work. Live work could become invisible when current facts changed. A correct repair to source data could alter the recorded shape of work already underway.
When correcting your source data is dangerous, the model is wrong, not the data.
The system was asking one mechanism to do two jobs: find current candidates and remember a past decision. A changing rule is good at the first job. It cannot safely do the second one by silently recalculating history.
That produced the principle at the center of the architectural change: A rule can propose a population. A decision has to be recorded.
The direction became clear. Rules could continue finding who qualifies now. The actual roster the business committed to work would be recorded. Later additions and removals would be explicit and traceable instead of quietly rewriting what had happened.
Part of that capability already existed before I could fully name the flaw. Production use explained why it needed to become the architectural direction rather than just one supported option. The proposal survived the operating leader’s strongest counterarguments and became the agreed path. Broader adoption is still underway; I am not claiming the migration is finished or that every historical issue is solved.
The product taught other lessons the same way. One of my own data-refresh designs froze the application during a real leadership meeting; the update failed safely, I found the cause, and changed the design. In another case, automation nearly made the wrong completion decision until an operator caught the edge case and the rule was narrowed.
- 01Rule — Find who qualifies based on what is true now
- 02Proposal — Present that population for review
- 03Decision — Record who the business actually chose to work then
- 04History — Preserve that roster even as current facts change
- 05Later changes — Add or remove explicitly, with the reason recorded
What this proved
The motion became substantially more legible. The system became a working surface for operators and senior leaders involved in planning and execution.
Before, no one could keep up with what was happening. After, we could say: We know what is happening. We could also investigate why a number changed, which population produced it, whether the source was current, and what had changed upstream.
The strongest proof is not that I can build a large dashboard. It is that I can step into work I do not yet understand, make it legible, build around what I learn, and keep investigating until the system reveals something deeper than the original request.
When the evidence showed that one of my own foundational assumptions was wrong, I changed the architecture instead of defending it.