Software Development Methodologies have been at the core for solving problems with computer software. There are many reasons why a methodology is important, but mostly, because it gives projects a beginning, a middle and an end. Projects are like stories and as such, they can have good or bad endings.
Let’s define what a Software Development Methodology is (I will be broad to serve the purpose of this post):
“A Development Methodology is a repeatable series of steps followed in order to complete a project involving the development of computer programs.”
Development methodologies are particularly important when there is a team working on a project. If just one person is working on the project, then the methodology is simple: just crank the code out. But when more than one person are working together then the level of complexity also increases proportional to the number of individuals involved in the project. Additionally, after the number reaches critical mass (defined as the absolute number of people with the right skill sets assigned to the project), one more person increases the complexity in an exponential manner. This WILL derail the project and the more people are added, the worse the project will fair. The proverbial “Mythical Man Month”.
The way I like to organize teams is as follows:
• Developers: Lead by a development manager, this group is responsible for coding. There can be more than one development team working on different components, products, releases, etc.
• Integration and Release: Lead also by a manager, this team is responsible for engineering the release during its different stages and for completing the release altogether.
• QA: lead by a QA team leader, this group is responsible for the engineering of tools used for automating regression and basic use cases.
• User Acceptance Testers (UAT): Also led by a team leader, they are responsible for functional testing. In other words, they look at the functional spec and press buttons to verify the application works as expected.
I left out two very important roles, project management and architecture.
I am normally heavily involved in architecture. I have some very definite ideas on how to architect massive distributed systems and I am not yet willing to relinquish that responsibility, but when I do, architecture and design are more general management areas than production. They do ensure that the product is built right, but that is true across all products being built. Also, the responsibilities of an architect extent pass the boundaries of projects into integration with outside products and services.
Project management, even though it is an incredibly important function, I see it as external to development as well. Indeed, it does set the mood, pace and organization of development, but still, I feel very strongly that it is more a part of management and administration than development.
Incidentally, I have the best Project Manager working for me: Sue Christensen. She rocks. She is the best and to boot, she complements me very well, specially when we do not see eye to eye.
Similarly to the way I like to set teams up, I like to set up operating environments in the following manner:
• Development: This represents each one of the developers’ workstations. Each developer should have a scaled down version of the application. The code is kept locally for the purpose of development, but it MUST be checked into a code repository like CVS or SVN on a regular basis, provided the code compiles.
• QA: This is a separate environment where the code is deployed for load testing, regression testing and other automated tests. There are multiples of these environments so different releases or components can be tested simultaneously. These multiple environments are key to the pipeline.
• Staging: This environment is a facsimile of production but scaled down. It needs to accurately represent production configuration so similar problems are manifested and they can be addressed before the release goes live.
• Production or Live: This environment is SACRED. It is the environment that is used to operate the company.
I see software development as a manufacturing process. Do not be mistaken though, I am very much into innovation and I see it core to my functions and what, as a CTO, I bring to the party; but innovation should happen outside of production and brought into production as part of a product rollout. In a sense, it is similar to architecture, design and project management. They are all external to the actual production and manufacturing of a technology product.
But what is the pipeline?
The concept of a pipeline is to carry out no less than a release per week. Ideally releases would happen every day, except for Fridays, but definitely not less than one. Each of these releases can be a bug fix, enhancements or even new products. But for each release to be put into the release pipeline a process needs to be followed.
This process is simple. Let’s take a new product for example. The product development group is responsible for writing the business, which should explore and document the business logic for the given product. In a perfect world the business requirements will be closed and complete, which is a pretty way to say there should be no holes in the document. This document is given to the project management group for review.
The project management group develops the functional specifications, which are sent to the development group for review. Of course there are consultations between the product and the project managers as the functional specs are written. Once development has a chance to review, a Joint Application Development (JAD) session is conducted. All concerns are brought up at the JAD session and documents modified. After a quick get together and making sure all is good, development starts. In some organizations before development starts there is a very formal process where the documents are approved and signed-off. I think it is too formal a step and should be avoided.
Development should work towards quickly building a functional prototype. The prototype should be bare bones of what the final product will be, but it should start showcasing the beginnings of that product. As development proceeds new versions of the prototype are deployed into a QA environment for everybody to start playing with each new version. Each one of the prototypes cycles should take no more than a week. Sure, there will be some features that will take longer to develop, thus they should either get developed as part of the initial prototype; or in a different branch and merged into the release branch once ready.
There are some very tangible benefits to having these prototypes rolled out early:
1) Transparency. Everybody can see how the project is shaping up.
2) Expectation Management. If there is slippage, it will become apparent immediately. Also if a feature is less than expected, it will become apparent as well.
3) Market Deviations. If a product takes too long to develop the market it was going to serve might have changed. Early prototypes allow product managers and management in general to measure the developing product against the market and make the necessary changes without derailing the timelines too much.
4) Mood. Developers see their work materialize faster. They get happier and work faster.
5) QA. The QA team can start building and running automated test cases against the prototypes and provide early feedback to developers, minimizing the QA and bug fix time at the end of the project.
6) UAT. The UAT team can also start setting up the right cases and start exercising them so as to get readier when the time to execute comes. Also, UAT can double up as a documentation team during the early stages.
As the process moves forward more and more of the product is showcased through the prototype cycles, until one day it is ready for the final push and QA, UAT and development get the product into the pipeline and finally out the door.
In what I outlined above there is nothing new. Except, if you have noticed, there is an active role for everybody. Nobody is waiting. Release cycles happen quickly from the conception of the product through the final release. The philosophy is: “Do not chew more than you can swallow and if the piece is too big break it in smaller parts. And it you can not, then in parallel, pick at it one bit at a time.”
Develop small pieces, fast, integrate all the time and continuously test. The prototypes are key to this. Get QA in early, generating feedback along the way. Fix bugs early, do not wait until the end where there is a greater chance to have regression problems.
The same process can be applied to enhancements, except that if it takes longer than 2 weeks to develop an enhancement, then it should be considered a product release. Furthermore, there should be a team dedicated to bug fixes and patches where they crank a few bugs every day. The process remains the same, except in shorter increments of time and business requirements are replaced by bug reports.
The pipeline is the form that development takes once you have a high degree of parallelism in your technical organization. And if you can achieve a pipeline, then development will happen faster, with better quality, meeting your market on better footing and at the end, it should help you get ahead of your market. It does take a great deal of effort to set up the pipeline, but once in action, technology as a whole becomes easier to manage.