How I measure engineering effectiveness
The cost of a change is rarely inside its ticket. Find where the loop actually waits, measure that, and take the baseline before you change anything, because a figure reconstructed afterwards is the weaker one.
My principles
06 operating rules
The cost is rarely inside the ticket
The expensive part of a component was a meeting that happened before its ticket opened, and it was paid again on every component. Counting tickets would have shown the same number of components with a slightly shorter cycle time and called that the result. So the first question is where the cost actually sits, not what is easy to count.
Measure the loop before you change it
Where does work wait, where does quality drop, and which constraint keeps coming back. The answer usually points at a coordination cost, a decision boundary, or an ownership contract nobody wrote down, and that is the unit to measure. Then the change goes there instead of at the stage that was never slow.
Take the baseline before you change anything
I have now shipped two changes I believe in and cannot prove, because I was confident enough about the diagnosis to skip the before. A figure reconstructed afterwards, however honestly, is weaker than one measured at the time, and it stays weaker forever.
A dashboard that counts is not a dashboard that measures
Accepted suggestions, tokens spent, prompts issued, pull requests per engineer. Any of them would produce a chart, and none of them follow a change from intent to production or carry the cost of review, rework, and a year of operating the result.
Quality and cost travel with throughput
A throughput number on its own cannot say whether the system got faster or just less stable, and a change that raises speed and instability together has not paid for itself. So the three get read together, never separately.
Keep it out of individual performance reviews
The moment a productivity metric lands in someone's review it stops measuring the system and starts measuring how people respond to being measured. These numbers exist to find the constraint in a delivery loop, and that is the only thing they are good at.
What that looked like
02 cases
Refusing the number the tooling offered
Teladoc Health · 2025-2026
- Situation
- I had rolled out an AI workflow for cross-platform component integration and then had to say what it was worth. The available number was tickets, components integrated per engineer per month, before and after. But the expensive part of a component was never inside its ticket. It was a shared meeting, two hours or more with several engineers reconciling one specification against four shipped implementations, and it was paid again on every component.
- What I did
- I refused the ticket count. It would have shown the same number of components with a slightly shorter cycle time and called that the result.
- I put the workflow change where the cost was, so the model read the specification against the released implementations and proposed the contract before the meeting, and the engineers reviewed instead of reconstructed.
- I measured the one thing I could observe directly, which was the meeting.
- I labelled everything else a reconstruction, because I had never instrumented the baseline before changing the workflow.
- Outcome
- The shared discovery meeting fell from two hours or more to about thirty minutes, and that part I saw directly. The workflow figure, approximately 3x with around 120 developer-hours a month, is a user-confirmed reconstruction and is labelled that way everywhere it appears.
One workflow, one team, one measured figure and two reconstructed ones. Instrumenting the baseline first is the single thing I would do differently, and it is the reason that rule sits on this page rather than in a footnote.
The change I still cannot prove
ASKfm · 2016-2021
- Situation
- A feature shipped successfully and I still call its delivery a failure. Planned at two months, delivered in about six, with scope moving after the architecture was built around the earlier version of it. I was confident about the diagnosis and changed the release model on the strength of the argument.
- What I did
- I did not baseline release predictability or scope churn first, because I was sure enough about the diagnosis not to bother.
- I reported the outcome qualitatively afterwards, as clearer ownership and more honest delivery conversations, instead of dressing it up as a measured result.
- Outcome
- The change was right and I still cannot show a number for it. The only figure I am confident in is the one that started it. Two months planned, about six delivered.
The same measurement failure as the case above, at a different organisation and years earlier. The release-model change has its own page. What it contributes here is that I have made this mistake twice.