Perceived component failures often originate elsewhere in the system: Bell Labs found that 4 of every 5 wrong numbers were correctly dialed numbers handled incorrectly, i.e., the attributed failure was not in the component people blamed.
Failure attribution is a prerequisite for any optimization or eval loop. If you instrument and optimize the named component before tracing the actual failure path end-to-end, you can spend engineering budget on a subsystem that is not the bottleneck.
Before optimizing or retraining any module in a multi-stage pipeline, instrument the full request path to locate where failures actually originate, not where users or dashboards report them.
4 out of every 5 wrong numbers dialed were correct numbers incorrectly dialed
The discovery that 4 out of every 5 wrong numbers dialed were correct numbers incorrectly dialed.