Author: Rafiq Gemmail

Scrum.org recently published an article titled AI as a Scrum Team Member by its COO, Eric Naiburg. Naiburg described the productivity benefits for Scrum masters, product owners, and developers, challenging the reader to “imagine AI integrating seamlessly” as a “team member” into the Scrum team. Thoughtworks’ global lead for AI-assisted software delivery, Birgitta Böckeler, also recently published an article titled Exploring Generative AI, where she shared insights into experiments involving the use of LLMs (Large Language Models) in engineering scenarios, where they may potentially have a multiplier effect for software delivery teams.

Naiburg compared the role of AI tooling to that of a pair-programming collaborator. Using a definition of AI spanning from LLM integrations to analytical tools, he wrote about how AI can be used to reduce cognitive load across the key roles of a Scrum team. He discussed the role of the Scrum master and explained that AI provides an assistant capable of advising on team facilitation, team performance and optimisation of flow. Naiburg gave the example of engaging with an LLM for guidance on driving engagement in ceremonies:

AI can suggest different facilitation techniques for meetings. If you are having difficulty with Scrum Team members engaging in Sprint Retrospectives, for example, just ask the AI, “I am having a problem getting my Scrum Team to fully engage in Sprint Retrospectives any ideas?”

Naiburg wrote that AI provides developers with an assistant in the team to help decompose and understand stories. Further, he called out the benefit of using AI to simplify prototyping, testing, code-generation, code review and synthesis of test data.

Focusing on the developer persona, Böckeler described her experiment with using LLMs to onboard onto an open source project and deliver a story against a legacy software project. To understand the capabilities and limits of AI tooling, she used LLMs to work on a ticket from the backlog of the open-source project Bhamni. Böckeler wrote about her use of LLMs in comprehending the ticket, the codebase, and understanding the bounded context of the project.

Böckeler’s main tools comprised of an LLM using RAG (Retrieval Augmented Generation) to provide insights based on the content of Bhamni’s wiki. She offered the LLM a prompt containing the user story and asked it to “explain the Bhamni and health care terminology” which it mentioned. Böckeler wrote:

I asked more broadly, “Explain to me the Bahmni and healthcare terminology in the following ticket: …”. It gave me an answer that was a bit verbose and repetitive, but overall helpful. It put the ticket in context, and explained it once more. It also mentioned that the relevant functionality is “done through the Bahmni HIP plugin module”, a clue to where the relevant code is.

Speaking on the InfoQ Podcast in June, Meryem Arik, co-founder/CEO at TitanML, described the use of LLMs with RAG performing as “research assistant” being “the most common use cases that we see as a 101 for enterprise.” While Böckeler did not directly name her RAG implementation beyond describing it as a “Wiki-RAG-Bot”, Arik spoke extensively about the privacy and domain-specialisation benefits that can be gained from a custom solution using a range of open models. She said:

So actually, if you’re building a state-of-the-art RAG app, you might think, okay, the best model for everything is OpenAI. Well, that’s not actually true. If you’re building a state-of-the-art RAG app, the best generative model you can use is OpenAI. But the best embedding model, the best re-ranker model, the best table parser, the best image parser, they’re all open source.

To understand the code and target her changes, Böckeler wrote that she “fed the JIRA ticket text” into two tools used for code generation and comprehension, Bloop and Github Copilot. She asked both tools to help her “find the code relevant to this feature.” Both models gave her a similar set of pointers, which she described as “not 100% accurate,” but “generally useful direction.” Exploring the possibilities around autonomous code-generators Böckeler experimented with Autogen to build LLM based AI agents to port tests across frameworks, she explained:

Agents in this context are applications that are using a Large Language Model, but are not just displaying the model’s responses to the user, but are also taking actions autonomously, based on what the LLM tells them.

Böckeler reported that her agent worked “at least once,” however it “also failed a bunch of times, more so than it worked.” InfoQ recently reported on a controversial study by Upwork Research Institute, pointing at a perception from those sampled that AI tools decrease productivity, with 39% of respondents stating that “they’re spending more time reviewing or moderating AI-generated content.” Naiburg calls out the need to ensure that teams remain focused on value and not just the output of AI tools:

One word of caution – the use of these tools can increase the volume of “stuff”. For example, some software development bots have been accused of creating too many lines of code and adding code that is irrelevant. That can also be true when you get AI to refine stories, build tests or even create minutes for meetings. The volume of information can ultimately get in the way of the value that these tools provide.

Commenting on her experiment with Autotgen, Böckeler shared a reminder that the technology still has value in “specific problem spaces,” saying:

These agents still have quite a way to go until they can fulfill the promise of solving any kind of coding problem we throw at them. However, I do think it’s worth considering what the specific problem spaces are where agents can help us, instead of dismissing them altogether for not being the generic problem solvers they are misleadingly advertised to be.

Gojko Adzic, author of Impact Mapping and many other classic product, engineering and testing books, recently announced the preview release of Lizard Optimization. His latest book discusses how responding to product-misuse and marginal corner-cases of the few long-tail users helped drive customer retention, and accelerate product growth. Continuous Delivery author, Dave Farley, recently interviewed Adzic for the GOTO Book Club, to discuss how these unusual use-cases can drive delivery of a better product for all customers.

Talking with Farley about feature areas to target for new product growth, Adzic discussed honing in on the curious and hard-to-understand use cases of small groups who appear to be misusing a product. Adzic referred to these use cases as “lizard behaviours.” He took the term “lizard” from a 2013 blog by psychologist Scott Alexander about surveying small groups with marginally held beliefs, such as those believing in conspiracies about lizards. Adzic told his own Lizard Optimization story of pivoting Narakeet, a product originally designed to generate narrated videos of PowerPoint presentations. He said:

The tool started as a way to make videos from PowerPoint. There were some people constantly making blank videos and paying me for that. Which made no sense at all … They were going through hoops and extracting videos just to produce an audio track… I simplified the way for those people to create just the audio file… That tiny minority, which was 1%, overtime became more than 99%.

InfoQ’s Ben Linders recently reported on a FlowCon France talk given by Fabrice des Mazery, CPO at TripAdvisor, where he discussed the cost and profitability risks of over-focusing on adding product features “because it’s better for the user.” Des Mazery cautioned that it’s important to take a broader ROI-centric focus to product roadmaps, which balance user-centric evolution of the product with longer-term profitability. TripAdvisor achieved this by transforming delivery teams into investors, with caps placed on cost. In Narakeet’s case, Adzic shared that adding support for the marginal use-case of a text-to-speech feature had resulted in improved UX and a significant reduction in operational cost.

Citing The Value of Keeping the Right Customers, a 2014 Harvard Business Review article by Amy Gallo, Adzic expressed the importance of retaining even those customers with marginal use cases, as marginal retention has a “compounding impact” on revenue. Gallo’s original article reported that “increasing customer retention rates by 5% increases profits by 25% to 95%.” Adzic told Farley that driving product improvements for all users was important for customer retention. He claimed:

If you want to grow a product, you can either go and acquire new users, or increase the retention of your existing users. And retaining users is incredibly more profitable at certain points of the life cycle.

A repeated point made by Adzic in his conversation with Farley was that by targeting marginal-use cases, one can make simplifications and overarching improvements across the product for “all users.” Another example he provided was that of a bug report filed about his mind-mapping application, MindMup, not being usable on a smart fridge. While this was not a use case he’d planned for, Adzic shared that by improving the touch capabilities of the application, they delivered usability improvements for all users, and widened the customer base. He explained:

Lizard Optimisation is a systematic way of figuring out how people are misusing a product, and then using those signals to make the product better. By making a product better for people who are misusing it, we can make a product better for everybody.

Within the book itself, Adzic calls out that Lizard Optimization is suited to products “where you want to grow market share, reduce churn or increase revenue.” He also explicitly calls out that for a product which is already “very mature,” having reached a point where you no longer need to grow it, then it is better to “focus on consolidation, technical scaling and reducing operating costs.” Lizard Optimization is in draft and available on LeanPub.

Gergely Orosz, author of The Pragmatic Engineer Newsletter, recently published the article Measuring Developer Productivity: Real-World Examples, co-authored with Abi Noda, CEO at DX and co-creator of the DevEx framework. The article analyses Noda’s survey of engineering metrics used across a broad spectrum of well-known tech giants. Noda found that rather than wholesale adoption of DORA or SPACE metrics, the indicators in use included many context-specific qualitative and quantitative metrics. Noda and Orosz provided guidance for defining such metrics by working backward from the outcomes sought by enablement teams.

Noda wrote that he “interviewed teams responsible for measuring developer productivity at 17 well-known tech companies.” For the article Noda and Orosz focused on 4 scales of organisation size, selecting Google at 100K staff, LinkedIn at 10K, Peloton at <10K, and scaleups such as Notion and Postman in the sub-1000 category. The metrics used range from typical PR and CI metrics, to methodically-selected indicators at Google.

Noda observed that in practice “DORA and SPACE metrics are used selectively,” rather than being adopted wholesale. He wrote that while the survey revealed that “every company has its own tailored approach,” he believed that “any size (of organisation) can adopt Google’s overall philosophy and approach.” Noda wrote that Google’s approach involves selecting indicators based on three classes of measurement relating to “speed, ease and quality.” He wrote that “tensions” exist between these three dimensions, “helping to surface potential tradeoffs.”

Noda wrote that Google’s measurements use “qualitative and quantitative measurements to calculate metrics,” as this provides “the fullest picture possible.” Noda cites a range of information acquisition approaches used by Google, from satisfaction surveys to “measuring using logs.” He wrote:

Whether measuring a tool, process, or team, Google’s Developer Intelligence team subscribes to the belief that no single metric captures productivity. Instead, they look at productivity through the three dimensions of speed, ease, and quality.

Similarly, Noda and Orosz described how LinkedIn uses a combination of quarterly developer satisfaction surveys with quantitative metrics. Noda wrote about a range of metrics used by LinkedIn’s Developer Insights team to drive its mission to reduce “friction from key developer activities.” The indicators used by this team include CI stability metrics, deployment success rates, as well as P50s and P90s for build times, code review response times, and the time for a commit to go through the CI pipeline. Noda described how the team bolsters such quantitative metrics with qualitative insights, using the example of comparing build time with “how satisfied developers are with their builds.” Objective numerical metrics are also de-noised by LinkedIn, using a “winsorized mean”:

What a winsorized mean does is it says: figure out your 99th percentile and instead of throwing away all the data points that are above the 99th percentile, clip them. So if your 99th percentile is a hundred seconds and you have a data point that’s 110 seconds, you cross out 110 and you write a hundred, and now you calculate your (winsorized) mean that results in a more useful number.

Noda wrote that Peloton, representing the 3-4K organisation, has evolved from initially capturing “qualitative insights through developer experience surveys” to also incorporating quantitative metrics. For instance, objective proxies for velocity such as lead time and deployment frequency are used to measure speed. He wrote that Peloton’s metrics also included qualitative engagement scores, time to restore services, and code quality as measured by the percentage of “PRs under 250 lines, Line Coverage and Change Failure Rate.”

Discussing smaller “scaleup” organisation such as Notion and Postman, Noda wrote that these often focus on measuring “movable metrics.” He explained that this is a sensitive metric that enablement teams can “move by impacting it positively or negatively with their work.” An example of this is “ease of delivery.” Noda wrote that this metric reflects “cognitive load and feedback loops,” and can be moved as it captures how “easy or difficult developers feel it is to do their job.” Another common movable metric reported was the “percentage of developers’ time lost to obstacles” and friction. Noda wrote of the power of such metrics in communicating their value:

This metric can be translated into dollars: a major benefit! This makes Time Loss easy for business leaders to understand. For example, if an organization with $10M in engineering payroll costs reduces time loss from 20% to 10% through an initiative, that translates into $1M of savings.

Given the contextual nature of such engineering metrics, Noda recommends 4 steps for an organisation aiming to define its gauges:

• Define your goals in a mission statement, explaining “why does your dev prod team exist?”
• “Work backwards from your goals to define top-level metrics” based on speed, ease, and quality.
• Define “operational metrics” tied to “specific projects or objective key results” Eg. adoption rate of a particular developer productivity-enhancing service.

Using examples, Noda pointed out that the metrics selected should be created with the dimensions of “speed, ease and quality” in mind. He illustrated with the example that if the goal is to make it easy for developers to “deliver high quality software”, the resulting metrics included “Perceived Delivery Speed,” Ease of Delivery” and “Incident Frequency.”

Orosz and Noda’s article is a follow-up to Measuring developer productivity? A response to McKinsey, a previous collaboration with Kent Beck that challenged and examined Mckinsey’s Yes, you can measure software developer productivity. McKinsey’s article proposed what it called “opportunity-focussed” metrics, “to identify what can be done to improve how products are delivered and what those improvements are worth.” The article included a discussion of developer productivity metrics to layer “on top of” DORA and SPACE. McKinsey’s piece included recommendations to encourage leaders to optimise for the efficiency of individual developer performance; an example area being “noncoding activities such as design sessions.” The metrics proposed include tracking “contributions by individuals” and measuring “talent capability scores.”

Warning of the dangers associated with measuring individual productivity, rather than the outcomes delivered, Beck shared his experience of seeing such metrics being used to “incentivize (with money & status) changes in the measures”. He shared that while this can result in “behaviour change” it is also subject to gamification, and incentivising “creative ways to improve those measurements.” Beck and Orosz encouraged leaders to instead focus on measuring “impact” rather than “effort”. Beck specifically recommended that such metrics should not be used for anything more than continuous improvement feedback loops of those that are measured. He also warned of safety issues caused by misuse of metrics measuring individuals for anything more:

Be clear about why you are asking & what your power relationship is with the person or people being measured. When the person with power is measuring the person without, you’re going to get distortions … Avoid perverse incentives by analyzing data at the same level as you collect the data. I can analyze my own data. My team can analyze its own aggregated data.

Noda also warned that at the level of CTO, VPE or Director of Engineering if asked for developer performance metrics, it is better to ensure reporting is at an appropriate level. Noda recommended that this can be done by selecting metrics representative of “business impact”, “system performance”, and “engineering organisation” level “developer effectiveness.” An example being project-level metrics, “user NPS” and “weekly time loss” to friction. Noda advised senior leaders:

In this case, my best advice is to reframe the problem. What your leadership team wants is less about figuring out the perfect productivity metrics, and much more about feeling confident that you’re being a good steward of their investment in engineering.

In their response to the McKinsey report, Orosz and Beck, shared a poignant meme as a reminder citing Goodhart’s Law, which says “when a measure becomes a target, it ceases to be a good measure.”