Every service was green; the workflow was broken
Each component in an evaluation system can pass its tests while the complete workflow remains broken. One service emits a trace, another normalizes it, an evaluator scores it, and a later component persists or displays the result.
Where should confidence live when the failure occurs between those boundaries?
The arrows became part of the architecture
Unit tests prove that individual instruments are in tune. Integration tests prove they are playing the same song.
For distributed evaluation infrastructure, the real product is the contract chain:
producer -> transport -> normalizer -> evaluator -> storage -> consumerEvery arrow is a place where fields can disappear, defaults can change, ordering assumptions can fail, and error semantics can become inconsistent.
We tested journeys instead of endpoints
The valuable tests were scenario-based rather than endpoint-based. A scenario starts with a representative trace and follows it through multiple components until the final evaluation result is observable.
The suite checks more than success. It covers missing fields, partial traces, duplicate events, evaluator timeouts, unsupported versions, and retries. Fixtures are kept realistic enough to preserve the relationships between events while removing private data.
The key design is to assert on stable contracts rather than incidental implementation details. Tests should care that a failure category and correlation identifier survive the journey. They should not fail because an internal helper changed its function name.
More mocks gave us less confidence
Adding more mocks initially felt like greater isolation and therefore safer testing. In practice, excessive mocking recreated the same assumptions inside the test and allowed incompatible services to agree with their own fictional dependencies.
The hard balance is realism versus determinism. A full production clone is slow and fragile; a fully mocked graph proves very little. The useful middle is real serialization, real contracts, controlled external boundaries, and representative failure scenarios.
The release criterion I use now
In distributed systems, the seam is a first-class component. It deserves tests, ownership, versioning, and observability.
Release confidence improved not because every function became perfectly tested, but because the most important journeys were exercised across the places where independent components had to agree.