Bug Backlog Management

Bug Backlog Management is an integral aspect of any software quality process framework. It consists in ensuring that the bug backlog, one of the most crucial artefacts produced by testers, is consistent, formally correct, up-to-date, efficiently organised, easily comprehensible, dense, and, in general, conforms to relevant quality standards set by the organisation or project. It is a recurring activity, whose frequency is highly context-dependent. Typical frequencies are: daily or weekly.

Bug Backlog Best Practices

Absence of Error Masking

Supposing that proper bug backlog management increases the chances of bugs being tracked and solved and thus leads to higher throughput, one key benefit is that bugs have an overall lower chance of being masked by other bugs. In contexts where bug density is high, it might happen that crucial bugs remain undetected because other bugs mask them, making it either unlikely or even impossible to detect the interesting stuff. In the area of manual testing, it undermines tester focus, thus leading to potential human errors or high level of stress. Consequently, fixing bugs regularly and efficiently contributes to earlier and more reliable detection of remaining, potentially more critical bugs (e.g., when frontend bugs render it impossible to find edge cases in the business logic). 

No Duplicates

Bug backlog size and complexity may increase rapidly over time. There are backlogs that contain ten years old critical bugs. Even in projects with a strictly limited time frame, backlog size may increase so drastically that it becomes virtually impossible to keep an overview of ‘everything’. In the age of AI, duplicate identification still involves manual processes. In unmanaged or insufficiently managed backlogs, no tester or test manager will be able to reliably detect all duplicates. In long-running projects or business-as-usual contexts, changes of personnel or role shifts may intensify this problem. So, keep your backlog in order so that you are able to easily notice duplicates and eradicate them instantly. 

Decreased Technical Debt

Many software products suffer from technical debt and almost no software product is free from technical debt. Bugs and technical debt share some commonalities, and sometimes there are grey cases where it is not even clear how a given phenomenon should be classified. Bug backlogs that are not properly managed increase the likelihood of bug fixes and feature implementations being built on top of faulty code. This makes it increasingly difficult and cost-intensive to tackle remaining technical debt—for example, when technical improvements render previous bug fixes obsolete (temporary workarounds) or require rework on features (changed technical basis). Keeping technical debt low by properly managing your bug backlog helps you in reducing costs and excess work. 

Commonalities Between Errors

This aspect is closely related to the No Duplicates observation. A properly managed backlog makes is much easier to identify commonalities, patterns, and internal relations amongst bug reports. This, in turn, is a prerequisite for avoiding local optimisations and fixes. Moreover, it helps in avoiding future errors, as insights gained from the backlog can be utilised in any Inspect & Adapt implementation.

Bug Backlog Anti-Patterns

From my experience, there are four main anti-patterns that are common in the area of bug backlog management. Let’s look at each of them in turn.

Bugs as Symptoms

Stereotypically, a bug report describes a particular faulty implementation of a particular functional or non-functional element of the test object. This conforms to the canonical definition according to which a bug is a ‘deficiency in a work product where it does not meet its specifications’. However, an anti-pattern commonly observable in software development project, which is related to the Commonalities Between Errors phenomenon described above, is that people focus entirely of the scope of a given report. They do not see the forest for the trees.

This could lead to local bug fixes (see below) and missed opportunities:

• Finding commonalities between bugs could unearth architectural or technological difficulties.
• Getting rid of some arbitrary deviance does not scale well; maybe the initial root cause candidate wasn’t the actual root cause, after all.
• Insights gained from bug backlog management can be fed back to development; this helps preventing bugs in the future and is thus more sustainable and cost-efficient than just getting rid of some symptom.

Fixing Errors Locally

Local error fixes are a very common phenomenon where a developer fixes exactly what’s been specified in the report. Contextualising a bug as an element in the wealth of similar defects is a time-consuming task. With well managed backlogs, this comes as a byproduct and does not cost anything. Fixing a bug very locally entails the danger that the entire chain of reporting, fixing, confirmation testing, etc. is started all over again once a new behavioural deviance had been found. Besides proper backlog management, quality standards of bug reports established at organisational or project level are a proper means of making sure that fixes are at not too local.

A special variant of the Local Bug Fix pattern is closely related to the Implementation Against Testers pattern presented below. In software development projects, there are at least four distinct points in time when a bug can be found: (a) during development; (b) after implementation completion; (c) after product approval; (d) after go-live. The interesting case is (b). Here, testers find a deviation (compared to a given specification artefact) and report on it. Normally, this would mean that the corresponding project artefact (e.g., a user story) is returned to development once, then tested again, and then forwarded for further processing (approval, deployment, etc., depending on context). 

However, there are project contexts where this is a loop of unknown length. A tester would return project artefacts to development repeatedly, essentially doing some kind of post-implementation refinement and continuous testing. These processes are as counter intuitive as they are seldom, but they exist. It is very helpful to be aware of factors that favour such a pattern (especially their combinations):

• inexperienced developers,
• inappropriately sliced backlog items,
• unequal distribution of domain knowledge in team,
• complex business rules,
• complicated specification structure.

Bug Backlog vs. Story Backlog

When stories are very big and testing is organised according to the Implementation Against Testers pattern (see below), stories are not just part of a backlog, rather they have their own backlog. Developers responsible for a story would make progress by working on items from this local backlog. This has a variety of drawbacks, among them:

• accountability diffusion (‘story’ implemented by n developers),
• untransparent velocity (story backlog out of scope of estimation process),
• unpredictable completion of story (you don’t know when you’re done),
• high level of frustration (tester and developer alike),
• additional overhead (context switches of testers and developers),
• potentially lower quality (as testers find high number of low-priority issues).

Implementation Against Testers

The primary pattern enabling the two just mentioned phenomena is the following. Usually, developers implement against some form of specification. This can be anything—from formal specification documents to requirements lists to user stories to conceptions in the head of a product owner, and so forth. In the context of the Implementation Against Testers pattern, however, they only pretend to be working against a specification. In fact, they work against the—growing—story backlog created by testers.

In this scenario, the tester is the domain expert, while the developer is some form of consumer. ‘Good enough’ is what’s been approved by the tester, until then he will continue to create little (pseudo) bug reports, which the developer may them tackle individually. So, the tester will not only function as the developer’s sparring partner but also as some ersatz product owner who slices a big story into consumable units.

This can work. In fact, I saw with my own eyes that it works. But it is incredibly inefficient:

• Tester does job he is potentially not qualified enough to do.
• Tester’s performance might be evaluated against unfair criteria (he spends significant time outside testing duties).
• Developer gets corrections constantly (e.g., several hundred per story).
• Tester’s frustration is high (e.g., requirement to grasp entire business logic loaded onto him exclusively).
• Increased management pressure (e.g., stories remain ‘in progress’ for months).
• Inefficient refinement process (asynchronous, undocumented, ad-hoc, etc.).
• Plus, everything already mentioned in above patterns.

Bug Backlog Properties

A clean bug backlog is implied by a clean product backlog policy: In many contexts, a bug backlog, formally, is a part of an overarching product backlog. There, bug reports live like any other item such as stories, epics, technical debt, tasks, features, or improvements. But, of course, bug reports are different. Two of its delineating features I’ll pick up below. Typically, there are distinct processes in place that apply to the bug backlog but not to the product backlog. For example, the allotted time frame for processing bugs is typically defined in advance.

No matter whether bug backlog and product backlog are technically separated—e.g., by maintaining them in different tools—the backlog policy applied to the product backlog applies to the bug backlog alike. A clean product backlog (policy) thus implies a clean bug backlog. Here are some of the things to consider:

• Default items (e.g., stories) conform to a scheme.
• Items toward the top are more refined than those toward the bottom.
• The entire backlog is ordered.
• Equal items are of equal size (e.g., epics).
• Estimations are reliable and stable.
• Metadata are complete and up to date.
• Dependencies and tracing relations are transparent.

In a project setting where these guidelines, among others, are in place, chances are very high that also the bug backlog is tidy, especially if the latter is integrated into the former.

Generic vs. missing vs. precise estimations

Product backlog items are usually estimated regularly. In a standard Scrum-based setting, all items until the next sprint but one are already estimated, the ones beyond that not. As bugs are part of the normal development workflow, bug reports need to be estimated like any other item on the list. However, bug reports behave differently than ‘usual’ bug backlog items. For instance, analysing the root cause—and thus determining the exact effort for fixing the bug—is typically a major share of the effort spent and, by its very nature, non-estimated.

This is why many teams apply some generic estimation policy instead of estimating bug reports individually. One way is to not estimate bug reports at all. This might have some drawbacks, depending on the project context. For example, you would process non-estimated items during an iteration and thereby interfering with your iteration statistics. Thus, it is common to assume some static estimation, e.g. three points or S. This approach lacks the precision of individual estimations but is very efficient and ensures that at least some effort is accounted for when including bugs in an iteration. 

Responsible roles

No matter who is responsible for maintaining the bug backlog the most important thing is that this is defined and documented. The exact role depends on a variety of context factors such as company size, domain, maturity of the organisation, test team size, third parties, and project complexity. The main alternatives are:

• test manager,
• tester,
• product owner,
• project lead, or
• everyone.

They all have their advantages and disadvantages. Letting the test manager oversee backlog activities is the most straightforward approach but only possible in settings based on the division of labour. A project lead, on the other hand, would only be responsible in very small projects where people assume responsibility for more than one role.

As the product owner is responsible for taking care of the product backlog it seems natural to let him manage the bug backlog as well, especially when both structures are represented in just one technical artefact. This approach has two significant drawbacks. Firstly, product specialists are usually not experts in software testing and thus less likely to know the intricacies of bugs. Secondly, they usually spend a major share of their workday in meetings and asynchronous consultations, which makes it difficult for them to focus on bugs like testers or test managers. When everyone is responsible for maintaining the backlog chances are high that at least someone will take care. However, individual chances are significantly lower due to the bystander effect. Also, there might be inter-person variability in quality standards.