Iteration Patterns

Inception Iterations

In Inception, the top risks are often either business risks or technical risks. The dominant business risk early on is typically ensuring project funding. Thus, a proof of concept prototype is often the result of the inception phase. The proof of concept prototype either demonstrates key functionality or some essential technology.

The first iteration of a new product is usually the hardest. There are many new aspects a first iteration must achieve besides producing software: For example, putting in place the process, team-building, understanding a new domain, becoming familiar with new tools, and so on. Be conservative in your expectations about how much of the architecture you can flesh out, or the degree of usable functionality you can achieve. If you aim too high, you risk delaying the completion of the first iteration, reducing the total number of iterations, and hence decreasing the benefit of an iterative approach. The first iterations should be focused on getting the architecture right. You must therefore involve the software architects in the planning process of early iterations.

Elaboration Iterations

In Elaboration, the iterations focus on defining a stable architecture, on designing and implementing the essential behavior of the system and exploring the technical architectural issues through a series of architectural prototypes. "Architecturally significant" scenarios are sub-flows which exercise the architecture of the system in defining ways.

Construction and Transition Iterations

Toward the end of Elaboration, and during Construction and Transition, change requests (also known as Software Change Orders or SCO's) begin to drive the iteration process. SCO's result from the following:

• enhancement requests
• change requests whose scope goes beyond the individual package or class.
• changes in the iteration scope and objectives.
• changes in requirements either proposing that the requirements baseline be changed, or accommodating an accepted change to the requirements baseline.

These SCO's are balanced against the existing project plan, iteration plans, and the existing risk list. SCO's may cause the priority of requirements to be re-evaluated, or may drive the re-prioritization of risk. SCO's must be managed carefully, however, lest project control be lost.

During Construction and Transition, the focus is on fleshing-out the architecture and implementing all remaining requirements.

Iteration Strategies

Unlike the Waterfall model, where the entire system is considered at once, we only consider a portion of the functionality of the system in each iteration. During each iteration, a subset of the total system is analyzed, designed and implemented. The choice of what the subset should be and how deep to delve are critical to reducing risk in subsequent iterations. There are two basic strategies: Wide/Shallow and Narrow/Deep.

Wide and Shallow

In the Wide/Shallow strategy, the entire problem domain is analyzed, but only the surface details are considered. All Use Cases are defined and most are fleshed-out in great detail, to get a clear understanding of the problem at hand. The architecture is defined broadly as well, and the key mechanisms and services offered by architectural components are defined; the interfaces of subsystems are defined, but their internal details are detailed only where significant risk or uncertainty must be managed. Very little is implemented until Construction, where most of the iteration occurs.

The Wide/Shallow strategy is appropriate when:

• The Team is inexperienced, either in the problem domain or in a technology area (including methodology or process).
• Sound architecture is a key requirement for future capability, and the architecture is unprecedented.

The strategy has some potential pitfalls, however:

• The team can get trapped in analysis paralysis (the illogical feeling that unless the design is perfect, one cannot implement anything).
• Early results are often needed to build confidence and credibility; the longer the project team goes without producing something executable, the less confident they feel about their ability to do so.
• Not enough of the technical details and challenges of the architecture are exposed to get a sense of the real technical risks

Narrow and Deep

In the Narrow/Deep strategy, a slice of the problem domain is analyzed thoroughly. The Use Cases related to this narrow slice are defined and fleshed-out in great detail, to get a clear understanding of the problem at hand. The architecture required to support the desired behavior is defined, and the system is designed and implemented. Subsequent iterations focus on analyzing, designing and implementing additional vertical slices.

The Narrow/Deep strategy is appropriate when:

• Early results need to be demonstrated to overcome a dominant risk, garner support or prove viability.
• Requirements are continually evolving, making it difficult to completely define all requirements before starting detailed design and implementation work.
• The deadline is mandatory, such that getting an early start on development is key to successful delivery.
• A high degree of re-use is possible, enabling a greater degree of incremental delivery.

The strategy is not without drawbacks:

• There is a tendency with this strategy for each iteration to develop software that is vertically integrated but horizontally incompatible. This is sometimes referred to as the stovepipe syndrome, and it makes a system difficult to integrate.
• It is not well-suited to developing systems in a completely new problem domain or based on an unprecedented architecture, since a large part of the functionality of a system must be sampled in order to achieve a balanced architecture.

Lessons Learned from Experience

Generally, early iterations will have more of a Wide/Shallow flavor, while later iterations (where a stable architecture has been developed) tend to follow the Narrow/Deep strategy.

The first iteration is usually the hardest, since it requires the entire development environment and much if the project team to be in place. Tool integration and team-building issues add to the complexity of the first iteration. Focusing on the architectural issues can help to maintain focus and prevents the team from getting bogged down in details too early.

Hybrid Strategies

• Narrow/Deep strategy used in Inception

  Where exploitation of a new technology is essential to the fundamental viability of the project.  Many e-business projects require new technologies to be explored to a much greater depth than might be done traditionally.  The proof-of-concept prototype is still considered a "throw-away", and merely explores the viability of the project concept.

• Wide/Shallow strategy used in Inception

  This strategy is pursued to gain an understanding of the scope of the system, and to sample the breadth of functionality of the system to ensure that the architecture is capable of delivering the desired capabilities.

• Wide/Shallow strategy used in Elaboration

  This approach can help develop a sound architecture, with selective Narrow/Deep focus to address specific technical risks. In Construction, with a sound architecture established, the focus can return to Narrow/Deep, where functionality is developed and delivered in a series of integrated increments.

• Narrow/Deep strategy used in Construction

  Construction iterations are always Narrow/Deep, with teams working in parallel to develop and deliver the required functionality. 

Special Considerations for New Teams

New teams are typically overly optimistic at first with what they can accomplish. To counter this, and to avert potential morale problems which occur when actual results fall short of optimistic expectations, be modest in the amount of functionality that can be achieved in the first iteration. Try to build experience while creating a sense of accomplishment and project momentum.

If the development environment and/or methods are new to the team, reduce the functionality of the first iteration to a minimum. Focus on integrating and tuning the environment and becoming proficient with the tools, then ramp-up the functionality content in subsequent iterations.

Expected Rework

Rework is good, up to a point. One of the major benefits of an iterative development is precisely to allow mistakes and experimentation, but early enough so that corrective actions can be taken. However technical people in particular tend to 'gold plate' or redo work to perfection between one iteration and the next.

At the end of each iteration, during the iteration assessment, the team should decide what part of the current release will be reworked. Expect rework to be allocated among phases in the following percentages, relative to the total system:

• Inception, 40%-100% - this is where you may develop throwaway, exploratory prototypes
• Elaboration, 25%-60% in early iterations; less than 25% in later iterations, or for an evolution cycle.
• Construction, after the architecture baseline, 10% or less per iteration and 25% total.
• Transition, less than 5%.

Rework is inevitable. When no one sees the need for rework, you should be suspicious. This may be due to:

• Excessive pressure schedule.
• Lack of real test or assessment.
• Lack of motivation or focus.
• Negative perception of rework as being bad, waste of resources, or an admission of incompetence or failure.

Too much rework is alarming. This may be due to 'gold plating' or to an unacceptable level of requirement changes. A business case must be done to evaluate the necessity of some rework.

Note that we do not include work de-scoped from the previous iteration (because of the timeboxed approach to iteration management) in the category of 'rework'. The Project Manager has to include this de-scoped work in the pool of functionality from which to define the next iteration's contents. Obviously, such work will normally have high priority. The Project Manager should also discover and carefully consider the reasons for the failure of the previous iteration to achieve its planned goals. For example, although we do not advise the arbitrary addition of staff in a desperate attempt to meet a schedule, running a project chronically understaffed - while repeatedly making ambitious plans for each iteration - is not sensible either. It usually leads to poor team morale and an angry customer. The right balance has to be found, and estimation models such as COCOMO II (see [BOE00]) can help with this. With each iteration, a project builds a history of achievement - of productivity and quality. A strong indicator for a Project Manager, in planning the next iteration, is what was achieved in the previous one.

Level of Planning

When the first-cut iteration plan is complete, the team leads, perhaps in conjunction with the project manager, can refine it into a working plan at the task level. The included Microsoft® Project Templates (at the task level) show how this might appear. Note though that these working plans are derived from the iteration plan, they are not separately produced, independent work products. It is important - if the project manager is to keep control - that the working plans can be rolled-up to status the project manager's iteration plan.