Author: Ben Linders

Agile software development can be done at scale with the use of technology like self-service APIs, infrastructure provisioning, real-time collaboration software, and distributed versioning systems. Lean can complement and scale an agile culture with techniques like obeyas, systematic problem-solving, one-piece-flow and takt time, and kaizen. Fabrice Bernhard spoke about how their company uses technology with lean thinking for doing agile software development at scale at FlowCon France.

The agile manifesto doesn’t apply to large organizations, Bernhard stated. Leaders looking for principles to keep their culture agile while scaling their software organization will need to look elsewhere. And unfortunately that “elsewhere” is now crowded with options called “agile at scale”, many of which are very bureaucratic and therefore not in the spirit of the agile manifesto, he mentioned.

Agile can scale, Bernhard said; there are many examples of organizations that scaled while maintaining an agile culture. In the body of knowledge of lean thinking, they found the principles they were looking for to scale their organization while staying true to the agile manifesto.

In the book The Lean Tech Manifesto that Bernhard wrote with Benoît Charles-Lavauzelle, he explores principles, systems, and tools that lean thinking provides to extend the principles of the agile manifesto. He mentioned some examples:

• Value models, obeyas, and value streams, to scale “Customer Collaboration” by ensuring “Value for the Customer”’ becomes the North Star of the whole organization
• Systematic problem-solving with PDCA and 5S, supported by team leaders and enabled in our digital world by collaboration technology, to scale “individuals and interaction” and transform the organization into a “tech-enabled network of teams”
• Jidoka, dantotsu, poka-yoke, pull, one-piece-flow and takt time, to implement “right-first-time and just-in-time” and scale “working software”
• Standards, kaizen, skills matrix and communities of practice, to scale “responding to change” with “building a learning organization”

Bernhard mentioned that they felt that lean thinking didn’t fully explain how some large agile organizations were succeeding. He decided to explore how the Linux open-source project and its community scaled from 1 to 55,000 contributors, where they used technology to address the scaling issues that they faced along the way:

The first scaling crisis happened in 1996, when Linus wrote that he was “buried alive in emails”. It was addressed by adopting a more modular architecture, with the introduction of loadable kernel modules, and the creation of the maintainers role, who support the contributors in ensuring that they implement the high standards of quality needed to merge their contributions.

The second scaling crisis lasted from 1998 to 2002, and was finally addressed by the adoption of BitKeeper, later replaced by Git. This distributed the job of merging contributions across the network of maintainers and contributors.

In both cases, technology was used to reduce the amount of dependencies between teams, help contributors keep a high level of autonomy, and make it easy to merge all those contributions back into the main repository, Bernhard said.

Technology can help reduce the need to communicate between teams whenever they have a dependency on another team to get their work done. Typical organizational dependencies, such as when a team relies on another team’s data, can be replaced by self-service APIs using the right technologies and architecture, Bernhard mentioned. This can be extended to more complicated dependencies, such as infrastructure provisioning, as AWS pioneered when they invented EC2, offering self-service APIs to spin up virtual servers, he added.

Another type of dependency is dealing with the challenge of merging contributions made to a similar document, whether it’s an illustration, a text, or source code, Bernhard mentioned. This has been transformed in the last 15 years by real-time collaboration software such as Google Docs and distributed versioning systems such as Git, he said.

Bernhard mentioned that he learned a lot from how the Linux community addressed its scaling issues. And where the first agile methodologies, such as Scrum or XP, focus on a single team of software engineers, lean thinking has been battle-tested at scale for decades in very large organizations, Bernhard said. Anyone trying to scale an agile organization should study lean thinking to benefit from decades of experience on how to lead large organizations while staying true to the spirit of the agile manifesto, he concluded.

According to Camilla Montonen, the challenges of building machine learning systems have to do mostly with creating and maintaining the model. MLOps platforms and solutions contain components needed to build machine systems, but MLOps is not about the tools; it is a culture and a set of practices. Montonen suggests that we should bridge the divide between practices of data science and machine learning engineering.

Camilla Montonen spoke about building machine learning systems at NDC Oslo 2023.

Challenges that come with deploying machine learning systems to production include how to clean, curate and manage model training data, how to efficiently train and evaluate the model, and how to measure whether or not the model continues to perform well in production, Montonen said. Other challenges are how to calculate and serve the predictions the model makes on new data, how to handle missing and corrupted data and edge cases, how and when to efficiently re-train this model, and how to version control and store these different versions, she added.

There is a set of common components that are usually part of a machine learning system, Montonen explained: a feature store, an experiment tracking system so that data scientists can easily version the various models that they produce, a model registry or model versioning system to keep track of which model is currently deployed to production, and a data quality monitoring system to detect when some issues with data quality might arise. These components are now part of many MLOps platforms and solutions that are available on the market, she added.

Montonen argued that the tools and components do solve the problems for the systems they were designed for, but often fail to account for the fact that in a typical company, the evolution of a machine learning system is governed by factors that are often far outside of the realm of technical issues.

MLOps is not about the tools, it’s about the culture, Montonen claimed. It is not about just adding a model registry or a feature store to your stack, but about how the people who build and maintain your system interact with it, and reducing any and all friction points to a minimum, as she explained:

This can involve everything from thinking about git hygiene in your ML code repositories, designing how individual components of pipelines should be tested, thinking about how to keep feedback loops between data science experimentation environments and production environments, and maintaining a high standard of engineering throughout the code base.

We should strive towards bridging the divide between the practice of data science, which prioritizes rapid experimentation and iteration over robust production quality code, and the practice of machine learning engineering, which prioritizes version control, controlled delivery and deployment to production via CI/CD pipelines, automated testing and more thoughtfully crafted production code that is designed to be maintained over a longer period of time, Montonen said.

Instead of immediately adopting a bunch of MLOps tools that are more likely to complicate your problems instead of solving them, Montonen suggested going back to basics:

Begin with an honest diagnosis of why your machine learning team is struggling.

The largest gains in terms of data scientists’ development velocity and production reliability can be gained with a few surprisingly basic and simple investments into testing, CI/CD, and git hygiene, Montonen concluded.

InfoQ interviewed Camilla Montonen about building machine learning systems.

InfoQ: How well do the currently available MLOps tools and components solve the problem that software engineers are facing?

Camilla Montonen: Most big MLOps tooling providers grew out of projects started by engineers working on large language model training or computer vision model training, and are great for those use cases. They fail to account for the fact that in most small and medium sized companies that are not Big Tech, we are not training SOTA computer vision models; we’re building models to predict customer churn or help our users find interesting items.

In these particular cases, these ready-made components are often not flexible enough to account for the many idiosyncrasies that accumulate in ML systems as time goes on.

InfoQ: What’s your advice to companies that are struggling with deploying their machine learning systems?

Montonen: Find out what your machine learning team is struggling with before introducing any tools or solutions.

Is the code base complex? Are data scientists deploying ML pipeline code into production from their local machines, making it hard to keep track of which code changes are running in production? Is it hard to pinpoint what code changes are responsible for bugs that arise in production? Perhaps you need to invest in some refactoring and a proper CI/CD process and tooling.

Are your new models performing worse in online A/B tests compared to your production models, but you have no insight into why this happens? Perhaps you need to invest in a simple dashboard that tracks key metrics.

Having a diagnosis of your current problems will help you identify what tools will actually solve them and help you reason about tradeoffs. Most MLOps tools require some learning/maintenance/integration efforts so it is good to know that the problem you are solving with them is worth these tradeoffs.

For one company, large language models created a breakthrough in artificial intelligence (AI) by shifting to crafting prompts and utilizing APIs without a need for AI science expertise. To enhance developer experience and craft applications and tools, they defined and established principles around simplicity, immediate accessibility, security and quality, and cost efficiency.

Romain Kuzniak spoke about enhancing developer experience for creating AI applications at FlowCon France 2024.

Scaling their first AI application to meet the needs of millions of users presented a substantial gap, Kuzniak said. The transition required them to hire data scientists, develop a dedicated technical stack, and navigate through numerous areas where they lacked prior experience:

Given the high costs and extended time to market, coupled with our status as a startup, we had to carefully evaluate our priorities. There were numerous other opportunities on the table with potentially higher returns on investment. As a result, we decided to pause this initiative.

The breakthrough in AI came with the emergence of Large Language Models (LLMs) like ChatGPT, which shifted the approach to utilizing AI, Kuzniak mentioned. The key change that LLMs brought was a significant reduction in the cost and complexity of implementation:

With LLMs, the need for data scientists, data cleansing, model training, and a specific technical infrastructure diminishes. Now, we could achieve meaningful engagement by simply crafting a prompt and utilizing an API. No need for AI science expertise.

Kuzniak mentioned that enhancing the developer experience is as crucial as improving user experience. Their goal is to eliminate any obstacles in the implementation process, ensuring a seamless and efficient development flow. They envisioned the ideal developer experience, focusing on simplicity and effectiveness:

For the AI implementation, we’ve established key principles:

• Simplicity: enable implementation with just one line of code.
• Immediate Accessibility: allow real-time access to prompts without the need for deployment.
• Security and Quality: integrate security and quality management by design.
• Cost Efficiency: design cost management and thresholds into the system by default.

Kuzniak mentioned that their organizational structures are evolving in the face of the technology landscapes. The traditional cross-functional teams comprising product managers, designers, and developers, while still relevant, may not always be the optimal setup for AI projects, as he explained:

We should consider alternative organizational models. The way information is structured and its subsequent impact on the quality of outcomes, for example, has highlighted the need for potentially new team compositions. For instance, envisioning teams that include AI product managers, content designers, and prompt engineers could become more commonplace.

Kuzniak advised applying the same level of dedication and best practices to improve the internal user experience as you would for your external customers. Shift towards a mindset where your team members consider their own ideal user experience and actively contribute to creating it, he said. This approach not only elevates efficiency and productivity, but also significantly enhances employee satisfaction and retention, he concluded.

InfoQ interviewed Romain Kuzniak about developing AI applications.

InfoQ: How do your AI applications look?

Romain Kuzniak: Our AI applications are diverse, with a stronger focus on internal use, particularly given our nature as an online school generating substantial content. We prioritize making AI tools easily accessible to the whole company, notably integrating them within familiar platforms like Slack. This approach ensures that our staff can leverage AI seamlessly in their daily tasks.

Additionally, we’ve developed a prompts catalogue. This initiative encourages our employees to leverage existing work, fostering an environment of collective intelligence and continuous improvement.

Externally, we’ve extended the benefits of AI to our users through the introduction of a student AI companion for example. This tool is designed to enhance the learning experience by providing personalized support and guidance, helping students navigate their courses more effectively.

InfoQ: What challenges do you currently face with AI applications and how do you deal with them?

Kuzniak: Among the various challenges we face with AI applications, the most critical is resisting the temptation to implement AI for its own sake, especially when it adds little value to the product. Integrating AI features because they’re trendy or technically feasible can divert focus from what truly matters: the value these features bring to our customers. We’ve all encountered products announcing their new AI capabilities, but how many of these features genuinely enhance user experience or provide substantial value?

Our approach to this challenge is rooted in fundamental product management principles. We continuously ask ourselves what value we aim to deliver to our customers and whether AI is the best means to achieve this goal. If AI can enhance our offerings in meaningful ways, we’ll embrace it. However, if a different approach better serves our users’ needs, we’re equally open to that.

Much of the change we experience in our software organizations is coercive. Software engineers, architects, and sometimes even people in software engineering management roles feel they cannot spark change without formal authority, Eb Ikonne mentioned at QCon London 2024. To catalyze change, he suggested identifying allies, inviting people to participate in the change, and creating and sustaining engagement through storytelling.

Ikonne mentioned that people tend to believe they cannot initiate change in their groups without formal authority and power over others:

We are told that we must do X, Y, or Z, and it’s implied that the consequences for not going along will be negative. No one really cares to consider what we think about said change.

After a while, we believe that the change can only happen this way, Ikonne argued. We then proceed on a mission to accumulate positional power so we can also cause change to happen this way. In doing so, we continue the cycle of coercive change and perpetuate this belief.

Ikonne explained how he took on a managerial leadership role for a software engineering team, after some success as a software engineer, because he thought the only way to make things happen within the team was to force change on people:

I was fortunate to have team members respond negatively to my approach and not lose my job. The lessons I learned from that experience made me reflect on and challenge my beliefs about change.

That change must be done coercively is a tacit assumption many people hold deeply within software development organizations, Ikonne said. Hence, attempts to have software development teams adopt new practices, tools, or technologies are often coercive, even when people don’t recognize the coercive nature of the change they’re initiating, he added.

Ikonne stated that people who don’t have much formal authority and power-over others in the organization, which is often the case for many software engineers, architects, and similar roles, believe they cannot spark change in their group. Even people in software engineering management positions believe their ability to catalyze change is limited to the groups they control.

To non-coercively catalyze change in groups, you want to identify allies, invite people to participate in the change, and create and maintain engagement through storytelling regardless of where you’re an architect, software engineer, or some other role on the team, as Ikonne argued:

In my experience, multiple people think the group or organization will benefit from change. If you’re a software developer who thinks your team will benefit from adopting a new set of design patterns, look for others who think the same way.

Or maybe you’re a team lead and think there is a more effective way to discuss technical challenges. Identify teammates who share your opinion.

Ikonne stated, “When it comes to group change, if you want to go fast and far, you must go with others.”

Demonstrating expertise is a fantastic way to expand and grow your informal authority, Ikonne said. For people in a software engineering context, like software engineers and architects, this means developing subject matter expertise in one or more areas of your work:

Become someone that people go to when they have questions. Always be willing to help others.

Informal authority isn’t something you can demand from others, Ikonne said. People have to give it to you because they respect you and what you’re about. The better your relationships are with people, the better your chances of expanding your informal authority within your organization, Ikonne said. If you want to catalyze change non-coercively, you need informal authority, he concluded.

InfoQ interviewed Eb Ikonne about catalyzing change and expanding informal authority.

InfoQ: How do you catalyze change in software organizations in the absence of formal authority and power over people?

Eb Ikonne: It’s not as much about the absence of formal authority and power-over people as it is about not relying on these organizational resources to cause change.

I’ve always found other people who think we’d be better off making the changes I’m thinking about. Engaging other software engineers, architects, etc in the change endeavor and having them champion the change in the networks creates a cascade of change within the group.

InfoQ: How can storytelling engage people into change?

Ikonne: To give an example, I’ve shared with a software engineering team how another team facing similar challenges adopted new technical practices (for example, committing to the main branch) and how those practices helped them overcome their challenges. This kind of story inspires and engages.

InfoQ: How can tech people expand informal authority within groups and organizations?

Ikonne: There are several ways to expand your informal authority, but investing in developing healthy relationships with people is fundamental. If more people did this, i.e., focused on their relationships, our workplaces would be radically different. I firmly believe this.

Take the time to get to know the people you work with regardless of their position in the organizational hierarchy. Invite people to chat over tea or lunch. Talk about shared interests you might have. To borrow from Martin Buber, move beyond the transactional I-It relationships- seeing people as objects- to an I-Thou relationship that sees people for the humans they are.

Communicating quality gaps, holding space for good testing, and writing automation are some of the ways that testers contribute to software teams. According to Maaret Pyhäjärvi, we need to think about testing, not testers. Collaboration and having conversations between team members can result in valuable impact that changes the product and the experiences of our users.

Maaret Pyhäjärvi gave a keynote about the impact of testers and testing in software teams at HUSTEF 2023.

A way that testers contribute to software teams is by identifying and communicating the quality gap, the concrete examples of things missing between what we mean we want, what we say we want, and what we have, Pyhäjärvi said. Some people call this bug reporting, but that misses the truly valuable part, choosing to have a conversation, and choosing which conversation is timely and relevant, she added.

Testers can hold space for good testing in teams. Instead of pointing at problems, pairing and ensembling in particular can help software developers discover they can do testing, as Pyhäjärvi explained in an example:

I was working with a developer who was testing their own feature, and one day I decided not to say what to test and do. They looked at me and said “You’d want me to click here” and I said nothing, they clicked things choosing what they thought I would do, and found problems they had been missing.

She did not realize that sometimes we should do less to let people do more, and just point out the pattern, Pyhäjärvi mentioned. Realizing that she has more impact – in the moment and in the long run – through guiding developers to discovery, rather than discovering for them, is an unusual but valuable contribution, she added.

Writing automation that stays behind when you leave, and does the work, is also a valuable contribution of testers in software teams, as Pyhäjärvi explained:

Building for ownership, building for time when I am no longer around to test myself, and ensuring my contribution is not just getting bugs fixed by finding them, but also in keeping track of past lessons in an executable documentation format, with test automation.

Pyhäjärvi mentioned that through testing, we can change the product, change the experiences of our users, and become better testers and programmers:

I like to think it’s about testing, not testers, and we need the courage to bring conversations that create and support change.

Very few of our meaningful impacts come from a single person, Pyhäjärvi argued. It takes a village, or a pair and a ladder effect where you no longer know exactly whose idea fed the final implemented idea. Collaboration creates great results, Pyhäjärvi said. Collaborative credit is acknowledging all parties, not just the one who spoke the final form of the idea, she concluded.

InfoQ interviewed Maaret Pyhäjärvi about testing productivity and sharing stories.

InfoQ: How can we measure the productivity of testing?

Maaret Pyhäjärvi: The work of testing is like having a white paper with invisible ink; no one can tell you in advance exactly what you will need to discover. I use all kinds of approximations in my projects. After I have tested a little, doing a tour or survey of the application, I usually guess how many bugs are hidden and set that as a budget that I work against, and track our progress in uncovering those. Working against a plan, this allows me to update the end goal as I learn, but also show steps of progress. Similarly, but with more effort, I create coverage outlines.

I have to say though, that lack of productivity with testers shows up as noise (bothering others) and as silence (missing relevant results). Testing provides information at best. At worst, testing provides information people don’t want or care for. I call that kind of information noise, and noise bothers others.

InfoQ: Why is it important that testers share their stories?

Pyhäjärvi: Bugs aren’t relevant without their meaning. Meaning is conveyed with a story.

Improvements over time, teaching our colleagues something, learning from our colleagues – these are impacts best shared with stories.

InfoQ: What’s the testing story that you would like to share?

Pyhäjärvi: In one project, it took a bit of effort over the two years to turn away from planning test automation to implementing tiny bits of it, and to realize that time spent warning about the limitations of automation is time taken away from reaching success with it.

I used to say automation could not do everything. I explained the limitations of automation again and again. And I did less automation. Then I chose differently. Every time I felt like warning of its limitations, I chose to add some valuable work to it. As a result, I got useful and improved automation. It is still not perfect and needs more thinking from me, but if I had chosen to spend time on selling the idea that it could go wrong, I would not have learned how it could go right.

I think of this as a story about the choices that we have, but don’t recognize we have, since pushing through in times of big changes seems safer than changing things. I don’t think we would have done a good job testing without centering exploratory testing and documenting selected lessons in test automation.

According to Marijn Huizenveld, discipline is key to preventing accumulating technical debt. In order to be disciplined you should make it difficult to ignore the debt. Heuristics like fixing small issues immediately, agreeing on a timebox for improvement, and making messy things look messy, can help tame technical debt.

Marijn Huizendveld spoke about taming technical debt at OOP 2023 Digital.

To prevent the accumulation of technical debt and foster discipline, Huizendveld suggested using a highly visible area to show the debt inventory:

In the past, I’ve made enemies in the short term, both within and outside of the engineering community of organisations, by featuring the technical debt board prominently in the hallway. It was positioned in such a way that you could not get around it, literally. That is confrontational, because it shows where the organisation is failing, but all you do is make a problem apparent that is already there.

It takes some time to adjust for various roles that feel a responsibility towards the underlying causes, Huizendveld said. In the long term, high visibility results in frequent conversations about the tech debt inventory and what you are doing about it as a software organisation, he added.

Good technical debt is intentional, enables benefits for the organisation, and is controlled. Teams can use a disciplined approach for managing and repaying technical debt, Huizendveld said.

Not having any technical debt means liquidity, and liquidity means optionality, Huizendveld mentioned. People have a bias towards preventing losses, called loss aversion in bias literature. You can leverage this bias to provoke change.

Huizendveld provided some heuristics that have helped him tame technical debt:

• If you can fix it within five minutes, then you should.
• Try to address technical debt by improving your domain model. If that is too involved, you could resort to a technical hack. In the event that that is too involved, try to at least automate the solution. But sometimes even that is too difficult; in this case, make a checklist for the next time.
• Agree on a timebox for the improvement that you introduce with the team. How much time are you willing to invest in a small improvement? That defines your timebox. Now it is up to you and the team to honour that timebox, and if you exceed it, make a checklist and move on.
• Don’t fix it yourself, if it can be fixed by machines.
• If it is messy because you have a lot of debt, then make it look messy. Please don’t make a tidy list of your technical debt. The visual should inspire change.
• Only people with skin in the game are allowed to pay off debt, in order to prevent solutions that don’t work in practice.

Huizendveld suggested leaving alone areas that are stable. You don’t need to improve something that does not get changed, he argued.

InfoQ interviewed Marijn Huizendveld about dealing with technical debt.

InfoQ: How much technical debt do we typically see in software development?

Marijn Huizendveld: There was research done by Stripe Inc. in September 2018, described in the research paper The Developer Coefficient, which stated that “given an average workweek of 41,1 hours, developers spend circa 13,5 on technical debt, and including bad code it’s 17,3 hours.”

If you consider that the remaining time of 23,8 hours has to cover all the meetings and ceremonies that relate to planning work to be done, then you could infer that there is really not a lot of time left for actual work. Even if the numbers reported by developers on the impact of technical debt are exaggerated by 50%, it’s still a sizable amount.

InfoQ: How can we measure success in dealing with technical debt?

Huizendveld: There are a bunch of indicators:

• Team happiness should improve: track how people feel with a simple smiley rating, and use the answers to figure out if you’re on the right track. If people are frustrated because of the toil due to technical debt, then you know what to do. If people are happy because of the reduction of this toil, then you know doing more of it will bring even greater happiness.
• Unplanned work should go down: there is a reason why a shortcut is called a shortcut: it is not a thorough solution. In some cases, unplanned work is the consequence of shortcuts having a meeting with reality. This hinders the throughput of the team, because planned work has to be halted, to deal with the issues that are occurring in the production reality.
• Mean time to bug resolution should drop: there is a large category of technical debt that could be classified as technical debt in work processes. It’s the debt that hinders making changes to the system in the quickest way possible. For example, manual, error-prone steps to get the software into the hands of the users. When this category of technical debt is reduced, fixes are in production orders of magnitude quicker. On top of that, there is the fact that a lack of technical debt in the business logic makes it easier to address bugs, because the system behaves more predictably.
• And finally, your time to market should shorten: In the end, it is the cumulative effect of all of these small improvements that make the big improvement reality. Faster time to market is the compound effect of happy teams that can focus on the planned work without distractions, in a system that works for them, not against them.

According to Neil Turner, continuous discovery for product development is regular research that involves the entire software product team, and that can actively inform product decisions. Equating continuous discovery to weekly conversations with one or more customers can be misleading. Combining quantitative and qualitative research methods can help software teams gather data and understand what is behind the data.

Neil Turner spoke about how Redgate does continuous discovery for product development at Agile Cambridge 2023.

Turner defines continuous discovery for product development as:

Regular customer research;

By the team building the product;

In pursuit of a desired outcome.

The main pitfalls that software teams experience when attempting continuous discovery are chasing weekly targets, carrying out unfocused research, and neglecting other channels of customer insights, such as metrics and surveys, as Turner explained:

Many teams get fixated on hitting weekly targets, and on speaking to a certain number of customers a week. This can lead to poor quality research as teams focus on research quantity over quality.

Turner mentioned that software teams can end up carrying out unstructured and unfocused customer research for the sake of hitting their targets (even if they are self-imposed targets), rather than considering the most appropriate research to help inform their product assumptions and decisions:

Just because a team is speaking to their customers on a regular basis doesn’t mean that they are carrying out high quality customer research. Most teams will benefit from a mixture of quantitative and qualitative data and it can be risky to make decisions based on the insights from a few customers.

Combining quantitative research methods such as surveys and metrics, with qualitative research methods such as customer interviews, can help teams to gather data across their customers and to better understand what is behind that data, Turner said.

Not all teams at Redgate carry out continuous discovery, Turner mentioned. It’s an approach that works well with an established product, but it’s not a one-size-fits-all approach, he added. It’s very much a case of a team choosing the most appropriate research approach to take given what they need to learn and the work being undertaken. This choice of research approach will be driven by the product designer working in, or with the team, Turner said.

Teams at Redgate that do carry out continuous discovery will often do this via short bursts of customer research, rather than say running a session at a set time each week, Turner explained:

They might plan for 2-3 days of focused research a month. This makes it easier to plan, schedule, and prepare for customer research sessions. It also makes it easier to see trends as insights are collected over the course of a few days, rather than say having weeks between sessions.

Teams will use tools such as Calendly to help automate recruitment and will often share the responsibility of facilitating sessions, along with writing up notes, Turner said. For example, some teams have set up a rota so that there is less of a dependency on a designer running all the research activities.

Turner mentioned that teams carrying out continuous discovery have been able to establish a good cadence of exploring a problem space to identify opportunities, validating ideas being worked on (such as via prototypes), and getting feedback for features that have made it into a product. This supports a dual-track discovery and delivery approach.

Teams also have a better understanding of their customers and can better empathise with their challenges. After all, it’s one thing reading some feedback from a customer; it’s quite another to hear that feedback directly from the customer’s mouth, Turner concluded.

InfoQ interviewed Neil Turner about continuous discovery for product development.

InfoQ: What did your software teams learn from doing continuous discovery?

Neil Turner: Teams have learned that there is no set approach to continuous discovery and that it isn’t the answer to every research question. For example, we have a team at Redgate whose focus is early research and development. They are better placed to run more traditional upfront research, rather than continuous discovery. They tend to work on very early concepts and don’t want to be slowly drip fed customer insights. Instead, they will typically prototype a concept and get early feedback through blocks of customer research sessions.

Teams have also learned that continuous discovery takes a surprising amount of effort from the team. It’s not just a case of scheduling some sessions with customers and hoping for the best. Sessions have to be carefully planned, well run, and then properly analysed as a team. The benefits are worth the costs, but there are certainly costs.

InfoQ: What advice do you have for teams that want to start with continuous discovery?

Turner: My main advice for any team starting with continuous discovery is to level up their understanding of continuous discovery, to start small, and to adapt their approach.

Too many teams hear about continuous discovery, jump into a cookie cutter version of continuous discovery (because they don’t know enough to refine their approach), and then give up when it’s not delivering the wealth of customer insights they expect it to.

According to Ilian Iliev, software developers tend to forget to do things they do not have to think about every day, which can cause delays or impact the functionality of the product during a software project. To prevent overlooking something, he suggested starting early with automating deployment, setting up error logging, and using lists and reminders of things that were forgotten previously.

Ilian Iliev spoke about what software developers can do to prevent overlooking things at Dev Challenge Accepted 2023.

People tend to overlook things when developing a new software product or service because they don’t have to think about them on a daily basis, Iliev said. This can cause delays in your software project or it can impact the functionality of your product, or do both, he mentioned. Everything that you forgot during the planning phase means either pushing the deadline or cutting corners and releasing less than promised:

Imagine that two days before release, you suddenly realise that you have totally ignored the fact that there is existing data that is incompatible with the new version of your application and you have missed coming up with a migration plan. That may mean a few days of delay, or weeks of refactoring based on your specific circumstances.

Iliev recalled how one of their servers crashed in the middle of the night as a result of a leap-second handling bug in the OS:

The whole system was operational, except for the part that was processing the customers` orders. The problem was that we found out about it a few days later from a client calling and asking why the system still had not processed their order.

This was a hit both to the company’s reputation and also to our self-esteem as developers, because we had never thought about having 24/7 monitoring of that piece, Iliev said.

One of the things to focus on earlier from a deployment and operations point of view is to automate deployments, Iliev said. You do not need a complex CI/CD pipeline; of course those are awesome, but to start with maybe a simple script that just does all the steps could be enough, he mentioned. Just do not rely on people remembering to execute five different commands in a specific order, even if they are documented. Deployment should be a one-click or command thing, Iliev said.

For operations, availability monitoring and alarming are a must, Iliev argued. This is also true for proper error logging; if something goes wrong with your system you should be able to track down the root cause of it, he added:

Without proper logging, you end up guessing what causes an issue or even worse, not knowing about it until clients start complaining.

Iliev mentioned that saying that something is a bad decision is often interpreted as hostile and also does not bring value to the discussion. He suggested expressing concern that something may not work by pointing out specific reasons, which opens the possibility of identifying possible issues at an early level.

Software developers should not be afraid to ask questions outside of their department, Iliev said. In the end, all of our work is combined together so having a general understanding of all the other aspects is important for the better integration of all pieces in the end product, he added.

Iliev advised teams that when starting a new project, to think about everything that went wrong last time and make sure not to repeat it:

Think about how your product will evolve and how you are going to maintain the system. Keep good communication between the teams and make sure all stakeholders are aware of all the complexities and possible pitfalls.

There will always be things to forget or overlook but the less there are the better the result, he concluded.

InfoQ interviewed Ilian Iliev about how developers can prevent overlooking something.

InfoQ: What’s your advice to teams that are embarking on developing a new system or product?

Ilian Iliev: Using a checklist with topics that you should pay attention to during the process helps a lot. Split it into sections for each phase and use it both as a tool to verify the completion of certain steps, and also as a source of questions that will help you get a better understanding of the project needs. The benefit of those checklists is that they are quick to go over and help you avoid missing steps that will lead to future problems.

InfoQ: Can you give an example of how such a checklist can look?

Iliev: I’ve created a Github project with Project Checklists which contains drafts of the checklists that I use in architecture, implementation, deployment and operation, and evolution and maintenance.

Here are two checklist excerpts to inspire you to create your own:

Implementation:

• Documentation
  • Data models documentation – what different attributes represent
  • APIs documentation – what the endpoints do, what the expected inputs and outputs are
  • Business logic – why certain decisions were made
• Code quality – style guide, linters, review processes
• Versioning
• Stability – unit testing, end-to-end & integration tests, load testing

Deployment & Operation:

• Single command deployment
• Availability monitoring – are you sure your system is running?
• Metrics reporting – how to measure the system performance
• Error logging – operation logs should be easy to access and understand
• Noiseless services – the system should not generate unnecessary noise, e.g. logs, emails, notifications, as it may hide real problems

According to Erika Chestnut, software organizations can develop a culture of quality with a clear commitment from leadership, not only to endorse quality efforts in software teams, but also to actively champion them. This commitment and advocacy should manifest in data-driven decision-making that strikes a balance between innovation and quality, ensuring that we maintain the highest quality of existing offerings, as well as delivering high-quality innovation.

Erika Chestnut gave a keynote about developing a quality culture at Testing United 2023.

Strong quality leadership is essential for fostering a culture of quality throughout the software company, Chestnut mentioned. Effective communication and regular training on quality standards are crucial for empowering employees to champion a culture of quality.

Chestnut stated that a culture of quality encourages innovation, as employees feel supported and empowered to explore new ideas and solutions, knowing that the organization values high standards and continuous improvement.

Product innovation without emphasis on maintaining high product quality can compromise customer satisfaction and the software company’s reputation, Chestnut said. Quality efforts often suffer when they are not aligned with business outcomes, leading to their perceived lack of value.

A culture of quality within a software organization is characterized by a pervasive and steadfast commitment to excellence in every aspect of its operations. In such a culture, quality is not just a department or a set of procedures, but a core value that informs decision-making at all levels, as Chestnut explained:

Leadership consistently demonstrates and communicates the importance of quality, setting a clear example and expectations. Employees across the board understand and take ownership of their role in maintaining quality, feeling empowered and responsible for upholding high standards.

A lack of data-driven decision-making hinders the ability to maintain and improve quality over time, as data and analytics are vital for identifying improvement areas and tracking progress, Chestnut mentioned:

A data-driven approach underpins the quality initiatives, ensuring decisions are informed by accurate and meaningful insights.

Software organizations with a clear culture of quality often experience a significant boost in customer satisfaction and loyalty, Chestnut said. This often leads to increased market share and a stronger brand reputation.

Internally, a quality culture fosters higher employee morale and engagement. Employees take pride in their work and are motivated by the knowledge that they are contributing to a product or service of high standard, Chestnut mentioned. This often results in lower turnover rates and attracts top talent who are eager to work in an environment where excellence is valued.

InfoQ interviewed Erika Chestnut about developing a quality culture in software organizations.

InfoQ: How do we build a quality paved path?

Erika Chestnut: A paved or “defined” path to quality improves the culture of quality across your entire organization. It is a living document and collection of the best practices, procedures, guidelines, and tools that are unique to each organization. This document makes it easier for teams to produce high-quality outcomes as it aims to reduce friction, remove obstacles, and provide clarity for delivery teams.

When building a paved path, you want to collaborate deeply with stakeholders across the SDLC to elevate opportunities to infuse quality in all areas of your delivery flow, not just the development step.

InfoQ: How can we balance quality and innovation in software development?

Chestnut: It requires a strategic approach that acknowledges the interdependence of these two elements. The key is to integrate quality considerations into the innovation process from the outset, rather than viewing them as separate or competing agendas. This involves fostering a culture that values both creativity and attention to detail, encouraging teams to think innovatively while maintaining a focus on quality standards.

It’s crucial to establish flexible but clear guidelines that allow for experimentation and risk-taking, yet have robust quality checks and balances in place. Data-driven insights should guide both innovation and quality assurance decisions, helping to strike a balance where new ideas are tested and refined without compromising on quality. Ultimately, the goal is to create an environment where quality and innovation are not seen as trade-offs but as complementary forces driving the organization’s success.

InfoQ: What can be done to maintain a sustainable culture of quality?

Chestnut: Maintaining a sustainable culture of quality in an organization hinges on continuous leadership commitment and integrating quality into every aspect of the business. This effort involves regularly reinforcing quality values, celebrating successes, and learning from setbacks.

Key to this culture is actively engaging employees, encouraging their input and involvement in quality initiatives, and providing them with regular training and development opportunities.

In essence, a sustainable quality culture is fostered through a blend of strategic leadership, employee engagement, integrated processes, and continuous adaptation.

Chris Smith believes that the idea that teams with very stable membership are best is an instinctively-held, traditional view. There are advantages to having the same group of people stay together, especially in achieving a time-bound software development project. However, in a world where we increasingly see product or stream-aligned teams who own long-living software from creation through to delivery, operation, and ongoing improvements, then optimising for very stable teams is not the best idea, Smith argues.

Chris Smith, Redgate’s director of engineering, spoke about self-selection reteaming at Agile Cambridge 2023.

Very stable teams can suffer from having a limited pool of perspectives and ideas, and they can become very fixed and comfortable, perhaps complacent, in their approach to software development and how things should be done, Smith said.

Technical and domain experts can find themselves stuck in teams, Smith shares. They feel unable to move to a different team, because others in the team and deliverables are entirely reliant on their efforts. In these cases, it often feels easier for an expert to quit a company than change a team within one, which is not good for the overall organization.

Smith believes that if teams hold on to their members jealously, this can create a lack of opportunity for learning new technologies or domains, and tackling new challenges. Many engineers see adding on to or improving their skill sets and experience as a key part of their career development. Limiting that by restricting them to a single team for a long time can damage staff engagement and, again, can lead to employee attrition.

Individual teams can become silos of practice, information and ideas, becoming little isolated ecosystems, Smith adds. These teams are disconnected from the wider organisation, and struggle to align their approach with a wider group of people or work in concert with other teams.

Smith suggests encouraging people to move between teams at a reasonable cadence. This allows good ideas and practices to naturally move with them. Social links between former teammates also protect against the dysfunction of siloed teams.

Redgate has an annual reteaming process to create more diversity of thought and experience in their teams. Over each of the last 5 years, between a third and a quarter of engineers have chosen to move teams during reteaming. While the teams themselves have had to sacrifice some efficiency to support these changes, Smith feels the benefit to the company is worth it.

Healthy teams have an element of change and renewal, and having new people, with their varied experiences and perspectives, join an established group leads to improved outcomes for the organisation, Smith concluded.

InfoQ interviewed Chris Smith about why aiming for very stable teams might be a mistake.

InfoQ: What are the possible downsides of stable teams?

Chris Smith: I remember joining a new employer and being put onto a great team on my first day, which was full of the most talented people I had ever worked with. Those folks had worked together for a long time and they had been really successful in delivering a new product to market, but what I saw when I started working with them was surprising.

The software engineers in the team worked way ahead of the test engineers, moving on to develop new capabilities once they considered the last one “dev-complete,” while the test engineers struggled to keep up and ask for subsequent bug fixes. They were cargo-culting Scrum, doing a perfunctory job of ceremonies like sprint planning and retrospectives, but those sessions were delivering very little value. The architecture of the software was overly complex and the code opaque, plus people couldn’t really remember why things had been done the way they had. And so on.

These folks were brilliant and talented individuals, but they had significant issues in how they worked…. they had somehow grown to have a collective blindness to these issues, unable to even see the problems that were apparent to a less experienced, outsider like myself.

InfoQ: How might a leader address collective blindness in their team?

Smith: An antidote to collective blindness might be ensuring teams have “cognitive diversity”. Quoting Matthew Syed from his book Rebel Ideas, “Groups that contain diverse views have a huge, often decisive, advantage”. He states that by embracing cognitive diversity, fostering constructive debate, and harnessing collective intelligence, individuals and teams can improve their problem-solving, make better decisions and drive innovation in our rapidly changing world.

It’s this idea that pushes me to challenge the idea that stable teams are ideal. They might be best for the short-term, but in the longer-term ensuring team membership has a healthy turnover of people and introducing folks with new experiences and perspectives will result in better outcomes and a more performant organisation.