Measuring developer productivity without breaking trust
What a request for one focus group taught me about measuring engineers, and the practices I now bring to every metrics project.
At a glance
- Role
- Vice President, User Researcher, JPMorgan Chase; led all four studies
- Organization
- JPMorgan Chase, on JET, the firm's internal engineering platform. Research in 2023, shared as an internal talk in July 2024.
- Methods
- Four studies: three focus groups, eight interviews in the US, UK and India, and two dashboard concept tests, read against SPACE, DevEx, DORA and DX Core 4
- Outcome
- A one-page do-and-avoid checklist and eight practices for measuring developer productivity that held up across every study and match the published frameworks
Summary
Every engineering organization wants to know whether its developers are productive. Most reach for the numbers that are easiest to count: commits, pull requests, lines of code. My research on engineering-metrics tools showed what happens next. Engineers stop trusting the tool, then they start working for the number. This case study sets out the practices that held up across four studies and match what the published frameworks recommend.
What the team got
- A one-page checklist
- Six things to do and six to avoid when measuring developer productivity, drawn from the research and matched to the frameworks. Teams used it to review their own dashboards; the eight practices below are its longer form.
- A shared definition
- Productivity is not one number. Activity counts are never used alone to reward or penalize people. The definition the organization could hold its tools to.
- Evidence across four studies
- What engineers, managers and engineering leaders said about metrics tools, with the quotes, sorted by what each audience needed.
- A talk
- The findings and the practices, shared internally in July 2024 so teams building metrics products started from the same ground.
The work
Where it started
The first of the four studies was a request for one focus group on Dev360, a developer-activity dashboard on JET, the firm's engineering platform. That study has its own page, Are we grading engineers?; the short version is that answering "what is developer productivity?" before running the sessions turned feedback on screens into a decision about purpose. The three studies that followed, on an engineering-metrics tool, a quarterly-review dashboard and a metric-distribution concept, asked the same questions of different audiences. This page is the synthesis.
What the field says
Researchers have been converging on the same answer for years: productivity is not one number, and activity counts should never be used alone to judge people.
- SPACE (Forsgren and colleagues, 2021) defines five dimensions: satisfaction and well-being, performance, activity, communication and collaboration, and efficiency and flow. It advises tracking several metrics across at least three dimensions, including at least one perceptual measure such as a survey, and warns that activity metrics "should never be used in isolation either to reward or to penalize developers."
- DevEx (Noda, Storey, Forsgren and Greiler, 2023) focuses on what developers experience: feedback loops, cognitive load and flow state.
- DORA measures software delivery, not people. Its five metrics since 2024 are change lead time, deployment frequency, failed deployment recovery time, change fail rate and deployment rework rate.
- DX Core 4 (DX, 2024) balances speed, effectiveness, quality and impact, and warns that speed metrics used alone can cause fear and counterproductive behavior.
In August 2023 the debate went public when a consulting firm proposed measuring individual developer productivity and prominent engineers pushed back that effort and output metrics invite gaming. Our focus groups had surfaced the same risk a month earlier.
What engineers told us
Across four studies in 2023, engineers, managers and engineering leaders said the same things in different words.
| Study | Who | What they said |
|---|---|---|
| Dev360, developer-activity dashboard | 3 focus groups, developers and managers | "Are we trying to grade engineers based on how many times they commit?" Commit streaks invite gaming; after-hours metrics reward overwork. |
| Engineering-metrics tool | 8 interviews, new, retained and former users in the US, UK and India | Unexplained gaps ("oops, data unavailable") and opaque calculations erode trust. Metrics should "sit on the side, observing, but not interfering with our work." |
| Quarterly-review metrics dashboard, concept test | 3 reporting leads | Worry about "vanity metrics." "Could I see the math that calculated this exact number?" |
| Metric-distribution concept test | 5 engineering leads, 5 lines of business | Statistical terms got in the way. Leaders wanted the median, the outliers, and which direction is good. |
What carries over
- Start with the decision, not the dashboard. Being asked for "a focus group" is a request for evidence, not a research plan. Ask who will act on a metric and what they'll do differently; a number nobody acts on is just a number people will be judged by.
- Measure teams and systems, not individuals. Managers in our groups valued seeing how a sprint went for the team. Nobody wanted a ranking. DORA and Core 4 are designed for teams and organizations.
- Never use activity alone. Pair activity with outcomes and with how developers experience their work, measured in a survey. This is SPACE's core advice, and it matches what our participants asked for.
- Choose words that describe, not judge. "Top contributor" and "outside business hours" made a dashboard feel like surveillance. "Engineering activity by week" says the same thing without a verdict.
- Show the math. Every metric needs its definition, source, freshness, and whether high or low is good. If data is missing, say why and who can fix it.
- Add context. Meetings, sprint phase and work that produces no commits all explain a quiet day. Without context, a chart invites the wrong story.
- Don't gamify. Streaks and leaderboards turn a measure into a target. Participants described exactly how they would game them.
- Tell people how the data will be used, and involve them. Say in the product who sees what and why. Let developers help choose the metrics. DX Core 4's guidance makes the same point about communicating how metrics are collected and used.
Sources
- Forsgren, N., Storey, M.-A., Maddila, C., Zimmermann, T., Houck, B., and Butler, J. (2021). The SPACE of Developer Productivity. ACM Queue 19(1).
- Noda, A., Storey, M.-A., Forsgren, N., and Greiler, M. (2023). DevEx: What Actually Drives Productivity. ACM Queue.
- DORA. A history of DORA's software delivery metrics.
- DX. Measuring developer productivity with the DX Core 4; Tacho, L. (2024). Introducing Core 4. Lenny's Newsletter.
- Orosz, G., and Beck, K. (2023). Measuring developer productivity? A response to McKinsey. The Pragmatic Engineer.