Tomato Sauce Marketing

Jump to Comments

This article examines some techniques for reducing the costs of software defects and thereby improving overall ROI. The view is from within corporate software projects and looks for some ideas from the world of open source software development and just in time inventory strategies.

The premise of these ideas is based on a theory that defects cannot be avoided nor easily controlled. However the time to resolve defects can. The idea that the longer defects remain open the more costly and difficult they become to control. This will be elaborated on in more detail below. Let us first examine open source software and JIT from a theoretical perspective.

Open Source Software

From Wikipedia, the free encyclopedia

Open source software is computer software whose source code is available under a license (or arrangement such as the public domain) that permits users to use, change, and improve the software, and to redistribute it in modified or unmodified form. It is often developed in a public, collaborative manner. It is the most prominent example of open source development and often compared to user generated content.^[1]

Although the above is the stricter definition of open source software Open source is a set of principles and practices that promote access to the production and design process for various goods, products, resources and technical conclusions or advice.

These principals have been used to good effect on many software projects and many of these projects exhibit lower and more rapidly solved defect lists. So the idea is to capture some of this value in attacking corporate built software and related defects.

In his 1997 essay The Cathedral and the Bazaar^[3], open-source evangelist Eric S. Raymond suggests a model for developing OSS known as the Bazaar model. This encapsulated a number of patterns;

Bazaar model should exhibit the following patterns:

Users should be treated as co-developers

The users are treated like co-developers and so they should have access to the source code of the software. Furthermore users are encouraged to submit additions to the software, code fixes for the software, bug reports, documentation etc. Having more co-developers increases the rate at which the software evolves. Linus’s law states that, “Given enough eyeballs all bugs are shallow.” This means that if many users view the source code they will eventually find all bugs and suggest how to fix them. Note that some users have advanced programming skills, and furthermore, each user’s machine provides an additional testing environment. This new testing environment offers that ability to find and fix a new bug.

Early Releases

The first version of the software should be released as early as possible so as to increase one’s chances of finding co-developers early.

Frequent Integration

New code should be integrated as often as possible so as to avoid the overhead of fixing a large number of bugs at the end of the project life cycle. Some Open Source projects have nightly builds where integration is done automatically on a daily basis.

Several Versions

There should be at least two versions of the software. There should be a buggier version with more features and a more stable version with fewer features. The buggy version (also called the development version) is for users who want the immediate use of the latest features, and are willing to accept the risk of using code that is not yet thoroughly tested. The users can then act as co-developers, reporting bugs and providing bug fixes. The stable version offers the users fewer bugs and fewer features.

High Modularization

The general structure of the software should be modular allowing for parallel development.

Dynamic decision making structure

There is a need for a decision making structure, whether formal or informal, that makes strategic decisions depending on changing user requirements and other factors. Cf. Extreme programming.

The first pattern also known as Linus’s Law has obvious use in defect control. Linus’s Law according to Eric S. Raymond states that “given enough eyeballs, all bugs are shallow”. More formally: “Given a large enough beta-tester and co-developer base, almost every problem will be characterized quickly and the fix obvious to someone.” The rule was formulated and named by Eric S. Raymond in his essay “The Cathedral and the Bazaar “.

In contrast, Raymond claims that an inordinate amount of time and energy must be spent hunting for bugs in the Cathedral model, since the working version of the code is available only to a few developers.

Just In Time

Just In Time (JIT) is an inventory strategy implemented to improve the return on investment of a business by reducing in-process inventory and its associated costs. The process is driven by a series of signals, or Kanban that tell production processes when to make the next part. Kanban are usually ‘tickets’ but can be simple visual signals, such as the presence or absence of a part on a shelf. When implemented correctly, JIT can lead to dramatic improvements in a manufacturing organization’s return on investment, quality, and efficiency.

A related term is Kaizen which is an approach to productivity improvement literally meaning “continuous improvement” of process.

Kaizen is a daily activity whose purpose goes beyond improvement. It is also a process that, when done correctly, humanizes the workplace, eliminates overly hard work (both mental and physical), and teaches people how to perform experiments using the scientific method and how to learn to spot and eliminate waste in business processes.

Kaizen must operate with three principles in place: process and results (not results-only); systemic thinking (i.e. big picture, not solely the narrow view); and non-judgmental, non-blaming (because blaming is wasteful).

How can these ideas be used?

This paper specifically attacks the problem of defect costs and control. It should be fairly obvious that as defects pile up from the field and remain in queues (which we will call inventory) – they exhibit similar characteristic to widgets on a factory floor. In other words they are costly to have lying around. It is well known that as software ages it become more difficult to maintain, mainly because the original developers have moved on to other things.

This is why open source code is valuable. However, in reality, and especially in corporation this is often impractical. Interestingly the actual source code is often only a small percentage of the actual how and why the software works i.e. the ‘sauce’ that makes up the project. So perhaps we should call it open sauce. This means making the ingredients of the project easily, quickly and efficiently available to as many ‘eyeballs’ as possible.

In essence we are proposing to find a process that takes defects from reporting to solution as rapidly as possible with as few layers as possible. Our solution calls for having as few people handling the defects as possible when moving from user to developer and then back into production.

Furthermore our goal is to deliver the software ingredients in a way that as many eyeballs can quickly and efficiently review it, comment and add value to a solution. Borrowing from JIT, we aim to reduce the time between defect reporting and resolution as much as possible. We do this using signaling method where users can easily build visual tickets of the defect.

Building more on the Kaizen approach, some of the principal ideas are non-judgmental, non-blaming, a systematic process and measuring results. We will now attempt to define a process that uses visual signaling to reduce the layers between defect reporting and solution.

Software bugs

Bugs are a consequence of the nature of the programming task. Some bugs arise from simple oversights made when computer programmers write source code carelessly or transcribe data incorrectly. Many off-by-one errors fall into this category. Other bugs arise from unintended interactions between different parts of a computer program. This happens because computer programs are often complex, often having been programmed by several different people over a great length of time, so that programmers are unable to mentally keep track of every possible way in which different parts can interact. Many race condition bugs fall into this category.

The Cost of Defects

Defects are costly from a number of perspectives. Clearly they have the potential to impact deadlines and productivity and create a roll through effect into other activities. Defects also undermine the future ability to deliver software solutions. Users react by creating barriers and resisting future moves. Frustration and dissatisfaction with processes soon follow.

These costs are often difficult to quantify, however there impact is large and can be wide spread. Trust, believability and confidence are often shattered. Future optimism is dealt severe blows.

Actually most clients accept that defects are simply part of the software process, there is little debate over this. What users want is fast, speedy and permanent resolution. There is nothing more disconcerting as defects which reappear time and again.

Reducing the Layers (thinning)

Defects can’t be avoided. They can be reduced from the outset of a project with proper project management techniques. However, they will always be present and part of budgets and cost controls. The question becomes how can you reduce the cost of maintenance, support and defect repair.

As described earlier the basis is to reduce the layers that defects need to move through in order to find a solution. In other words – is it possible to allow an end-user to describe a problem space and immediately submit it to development teams for resolution?

Today’s desktop technology and tools certainly permits this to be done. Thereafter a collaborative medium is required to that as many eyeballs can view the problem and stimulate input.

The solution is found in the rapid rise of YouTube.com and Google video. Video is all the rage and for good reason. Video encapsulates the problem and describes its nuances easily. Then if this video of the problem space can be routed for comment rapidly and efficiently until it reaches the development team, it is likely that the defect will move rapidly to solution without interference and unnecessary layering. Get enough ‘eyeballs’ to review and comment on the problem space, and your developers will probably resolve the issue fairly rapidly.

Some of the benefits of this approach are that it enables parallel development because it encourages modularization of defects. It accepts that defects are normal and avoids finger pointing and even encourages a more rapid deployment with new releases more frequently.

Process and Results

Let us revert back to the notion of open source models. The open source model of operation can be extended to open source culture in decision making which allows concurrent input of different agendas, approaches and priorities, in contrast with more centralized models of development such as those typically used in commercial companies.^[2]

Counting defects as a goal is perhaps rather pointless. Since defects are a symptom of the original project team, a better approach is to measure the speed or delays as defects are solved. To accomplish these goals build a process that encourages a culture which allows concurrent input that captures a visual representation of the problem.

Tools and Techniques

A number of tools such as GoToMeeting and Camtasia screen recorder make it fairly simple for support teams or even end users to produce videos of their problem space. Imagine a GoToMeeting session is recorded – between a support person and end user – where the screen results and commentary can be captured. Thereafter a simple means such as ftp can be used to transfer video to centralized servers. Then the smart work begins.

Server technology can be built which automates much of the process. Firstly from the uploaded files defect cases will need to be built. This becomes a collaborative document which can be routed for comment to what the group perceives as the ‘eyeballs’ needed. The key here is the video productions which are typically made in 20 minute segments and can be viewed using the simple paradigm that YouTube developed, namely embedded video browsing.

From the routing and commenting a collaborative piece begins to form. With experience this forms the basis of the input the development team can review and code. The very nature of the visual document makes it simple to send defects to offshore or near shore locations for developer interaction. Again the whole principal is to reduce the back and forth between layers. To move the defect rapidly from the problem source to solution with as few delays as possible. It encourages the idea of early release so more beta-testers are available for feedback.

Conclusion

Software defects, although a reality, are a huge hidden cost – often neglected and underestimated by project teams. However, they can often undermine end users confidence and satisfaction. Although end users often understand that defects will always exist, they demand better than perceived response times and solutions.

The best way to accomplish this is to remove the layers that delay defect resolution. This can be done using current desktop tools to produce screen and sound videos. This in itself is not enough. The entire defect needs to be encapsulated in a collaborative routing mechanism which rapidly collects eyeballs concurrently and in a decentralized fashion. The overriding objective is to reduce wait times and dependencies between decision makers and encourage more dynamic and fluid responses, yet documenting the inputs along the way.

This methodology should ultimately encourage rapid early releases, frequent integration, and several versions as seen in many open source projects. With a Kaizen culture of non-blaming and non-judgmental it is possible to stimulate corporate software development with some of the successful techniques found in the open source world.