subject: Uncertainty is certain: A case for agile development practices [print this page] Uncertainty is certain: A case for agile development practices
"You tell us what to build, and we will build it." It sounds pretty simple, and since 1970 the "waterfall" model of software development has been used on the vast majority of software development projects. From an engineering perspective, this seems to be a fairly direct and natural approach to solving a problem. First we will gather all of the information, and then we can begin construction. Once the product owner has conveyed everything that needs to be known, the team should have the skills to build it. This implicitly lays the problem squarely and the product owner's feet: Just make sure that you tell us exactly what you need, and make sure that we understand it. (and P.S. Make sure you get it right the first time.)
Any engineer can tell you a tale where the gist of a problem' begins where mid-way through the construction of the project the product owner(s) either develops new requirements or clarifies existing ones. Waterfall projects run through just a few phases. The areas we are discussing are the requirements phase, and the development or construction phase. With the requirements phase, the engineering team needs to gain a thorough understanding as to what is required. With the subsequent construction or development phase, the team builds the application(s) to their understanding. Several other steps may be involved depending upon organizational structure and requirements, but these two steps are the major focal area addressed by our software development methodology.
Due to budgetary constraints, each of these phases or steps will ultimately be "timeboxed", meaning you have a set period of time to develop requirements, and a set period of time to get the work completed. Often this is an unspoken compromise, and this in effect often constrains the project to meeting as many of the requirements as possible in the specified period of time. Typically, corporate budgeting and capital rationing require determining the cost of the project ahead of the requirements gathering, which in itself is a conundrum. With the cost set, and the functional requirements not completely thought through, all of these projects inherit some amount of un-quantified risk from the outset.
One of the key differentiation points with agile methodologies that may not be immediately apparent is the amount of time the product owner and engineering teams spend analyzing requirements. The engineering team spends the entirety of the requirements phase developing an understanding of the requirements, and also gets to clarify and develop their understanding through the construction phase as well. One key point is that the product owner had a much shorter time limit to develop their requirements, ideas and solutions : they were "timeboxed" into the requirements phase. One of the key benefits of agile, which I believe isn't always expressed, is that the product owner will have the same opportunity to analyze, digest and understand the requirements of the platform as the engineering teams the entirety of the project. The idea embedded into the SCRUM approach specifically, works to enhance the job of partnering the requirements owner with the engineering teams for the life of the project. Working together throughout the course of the project on requirements refinement becomes the proficiency of the entire team.
Over the years numerous techniques have been developed to address the deficiencies inherit in the waterfall method. Some commonly seen deficiencies by phase are:
For requirements engineering:
The product owner may fail to enumerate requirements
The engineering team may fail to fully develop or understand requirements
For the development cycle:
The development team may introduce coding error
The development team may not meet established schedules
While having participated in projects where the engineering team failed to fully understand the requirements of the product owner is certainly not a rare occurrence, there have been great improvements in this area through team-centric process improvement.
Addressing shortcomings
Waterfall projects traditionally develop some of the same ad-hoc remedies to deal with issues imposed by the timeboxing of the requirements and development, errors in requirements definition or understanding, and even to accommodate for coding errors. We often perform one or more follow on releases to address these shortcomings. These can be in the form of "Phase 2" releases, "enhancement" releases or just subsequent versions of the project. In doing this, these projects naturally follow the same waterfall process. This is repeated until everyone is satisfied, or we no long have resources available to do so. In our attempt to timebox different phases of the project, we planned on getting everything right the first time, without error. When we can accomplish this, it is a great success. Often, product owners are left waiting for "Phase 2".
Remediating the issue
One alternative approach is to simply remove the divide between the product owner and the engineering team. What if, through the entirety of the project, the product owner was an integral part of the engineering team? Project managers work towards getting the entire team motivated and headed in the same direction and more importantly interested in the success of the project. If everyone were tied to the overall success of the project, rather than the deliverable for a particular phase of the project, would this lead to greater commitment from everyone for the duration of the project? Modern methodologies such as Scrum do just this. Instead of attempting to gather and digest all of the requirements at once, the team can now take them a piece at a time. Anything that is not yet fully understood sits on the shelf while the team works through issues that it is prepared to manage. This accomplishes a few things:
It allows understood features to be built and delivered. Whether the end user can make use of parts of the new functionality, or simply by getting features into testing sooner, progress is more easily discernable and monitored.
It allows the product owner and engineering team to work through issues together. Immersing the product owner into the cycle, keeps visibility on issues that need to be ironed out with requirements, and just as importantly it helps the product owner to work through requirements issues that he may not be able to articulate in the first month of the project. Increasing the time allowed to articulate requirements from a beginning phase of the project to the life of the project will undoubtedly increase accuracy and overall understanding.
It gets the whole team into a cycle of producing requirements, developing changes, and implementing them. With this repetition comes competency. Greater efficiencies will be developed in working through tough issues a piece at a time.
The difference that matters
With the waterfall method, we are laying the project out into distinct phases, where we time box requirements and development. Hoping to get everything right the first time, and when we don't -- repeating this process with either subsequent phases or enhancement releases until the project requirements are satisfied. In this case, we were planning on no requirements changing over the course of the project and getting everything correct the first time. With an agile methodology, such as Scrum, we are planning on things changing, and the complexities of the project are allowed to dissolve over the course of the project. The latter method simply manages what the first did not, change. By planning on requirements evolving over the course of the project, our methodology better reflects what we expect to happen and makes agile methodologies more consistent with our goal to manage the effort effectively.
