Thinking about Quality

Another interesting consequence of adopting agile methods, is that by pushing ownership of much planning and tracking downwards to the engineers, it removes a fair amount of work that used to be done by managers. Decreasing micromanagement certainly increases management bandwidth. I've noticed that since we switched to agile, 1-1 meetings with my managers have gotten shorter. A lot of the stuff I used to bug them about in person, I can just look up in VersionOne, and it's on its own with no prompting on my part in much better shape than it ever was before. This has been observed elsewhere too. Now that, besides being cool in and of itself, has an interesting implication.
Namely, it seems that agile methodologies encourage flatter management structures than traditional waterfall oriented approaches. You can either have your managers do more genuinely useful work, or simply have less managers. This idea bears more exploration.

A lot of times agile development methods are characterized as applying mainly to small teams. This is definitely not true. It is true, that a process like XP can only meaningfully done using a fairly small team. However, this is not the end of the story. The Scrum methodology lends itself very well to organizing large efforts as collections of teams, whereby each team practices scrum, and then the teams interact via a scrum-of-scrums. This mechanism is quite scalable. It also lends itself to interesting extensions. One of the useful qualities of Scrum is that it is quite limited in what it deals with, primarily project management. It has little to say about specific technical matters. The reason this matters is that in any fairly large organization there is likely to be a range of problems and a range of teams tackling them. These teams may well have different approaches and attitudes. Mandating a single methodology makes the system quite top-down and somewhat violates the self-organizing spirit of agile methods. The way out is to use Scrum as a general organizing principle around the organization, as a minimal set of practices that everyone uses. In our case that entails everyone using VersionOne in a consistent way, scheduling iterations on a common rhythm, and we have our top-level scrum meeting. Beyond that, each team, and sub team, is allowed considerable flexibility as to how they want to approach their piece of the puzzle. This means that different teams can adopt slightly different methodologies. The scheme works well because it allows people to make appropriate choices for themselves, while still ensuring common interfaces to allow people to cooperate properly. It is therefore both popular and efficient.
This mechanism also illustrates another misunderstanding about agile methods, namely that they are they are somehow opposed to CMM style approaches. In fact, our methodology, as spelled out in our Wiki, constitutes our defined process. The individual processes followed by each team are the tailored processes. The metrics in VersionOne and in our bug tracking system provide the level 4 metrics and feedback loops for CMM. The important point is that one should not understand CMM as saying that EVERYTHING should be documented. Think of the physics of gases. I can tell you with great detail about the relationship between the pressure, volume and temperature of a gas, but I can't tell you the exact position or velocity of a given gas molecule. CMM needs to be interpreted the same way. The processes we agree upon and write down are the general principles we can fully sign up for and follow exactly. We simply don't talk about those things which are variable and not part of our model. This is the key to successful processes: only define the minimal principles which make your world function; everything else is a waste of time. To achieve CMM level 3, you need to define processes for everything worth agreeing upon, and no more than that. Ockham's razor should be liberally applied in this respect. Keeping the processes simple, in turn makes it easier to measure and learn. The beauty of the VersionOne tracking system is in fact that it provides the good metrics and feedback one needs to improve. In this respect, CMM level 4 is dependent above all on having a solid set of processes, and good tooling support.
What about repeatability ? CMM implies a repeatable process. And we note that the agile processes are highly repeatable. Once the teams get good at it, you can do one iteration after another. Notice again, that it's not the particular actions of each programmer that are repeatable, it's the aggregate behavior of the team. Also notice, that repeatability does not imply a mechanical determinism. Risk and uncertainty always remain a constant in innovative software development. The approach of always tackling high risk and important tasks first, and decomposing large project into incremental deliverables is in a macroscopic sense very reliably repeatable, however, the individual twists and turns of specific projects still fluctuate as randomly as the molecules in a gas cloud.
In that sense agile methods do for software development much what physics did for the study of natural phenomena, in that allows one to abstract away from the particular to the general, and to understand the common patterns underlying apparently unpredictable details. This point was driven home to us when our senior VP was startled when one of our teams became totally stuck on an incompatibility with a new release of a third party product. He said, but I thought Agile would prevent this ? Of course not! Agile methods give predictable ways for managing risk, but they do not in any way control or remove the underlying risk. As is often said, the devil is in the details. The details are why software development is hard. But recognizing where we can be precise and where we need to be flexible is the key to successfully organizing software development.

As everyone knows, while eggs are quite fragile, it is still much, much easier to keep an egg intact, than it is to unscramble an egg after it's been scrambled. The same simple principle really applies to software to. If one finds that one is spending too much time and effort on QA, one has to ask oneself, why is this egg so scrambled ? How do we keep it from getting scrambled, so we won't have to unscramble it later ? When we first adopted agile methods, there was much fear in QA that they would become completely overworked, as they would have to do their normal work, PLUS all this continuous regression testing. In fact, the opposite has happened. By working with development to keep the system alive and well at all times, they have managed to prevent the deep scrambling that usually requires most of their attention. In fact agile approaches give QA more time to do actual productive testing, and spend less time trying to untangle a poorly integrated mass of half-finished code. This makes QA faster, yet actually more thorough, and indeed improves morale quite a bit. The realization is that our old waterfall oriented process tended to cause large integration problems that consumed a large part of the putative QA period. This in turn required QA to work lots of overtime on an already long schedule to compensate. With our new approach, the system is always integrated, and therefore testing is both easier to schedule, and eats up less time.

So I'm not the only one who's thought about the connection between Heidegger's thought and software development. Michael Hamman's blog on Humans and Computing has a really neat article on the application of Heidegger's concept of breakdown to software development. In particular, how successful problem solving requires us to break through the limitations in our world view imposed by our current conceptualization, and in better aligning our selves with the underlying real world reconstitute our conceptualizations. This phenomenon which is central to all problem solving, occurs over and over again in software development. How often is someone stuck on a bug, and they can't see it and can't see it, and suddenly they realize that if they look at things slightly differently the problem is obvious. The trouble of course is that what we think is real, is simply what we think is real. Sometimes we just need to stop thinking long enough to find out what is really real, so we can improve the reality of what we think.......

Part of our company's transition to Agile Processes has involved a determined push to move ownership of task schedules from managers to engineers. Whereas in the past managers would construct schedules, tell people their assignments, and check to see that they were doing them, in our new world, the managers only construct the stories, and leave defining the tasks, and indeed editing and improving the stories, to the engineers. At first people were annoyed, because they were being expected to make written commitments as to how they would spend their time. You mean I have to manage my time ? Yes. The public scrutiny imposed by our VersionOne system makes it very clear what people are doing and not doing, and leaves little opportunity for procrastination or disorganization. So people have had to adjust to making public commitments as to their plans, and documenting their activities.
On the other hand, some remarkable things have happened. For one thing, many people who used to be very passive about their work, relying on their manager for all instructions as to what to do when, have actually taken responsibility for their work, actively defining tasks, tracking them, and on their own coming to their manager to announce updates. This in turn has started to unburden some of our managers, as they don't have to check for status so much. Things are starting to happen on their own. One on one meetings are becoming shorter in a number of cases.
Another interesting phenomenon is the formation of virtual teams of developers and QA working together. In a waterfall system like we used to have, engineers would often finish their work weeks before QA would look at it, and then have to return to it. With everything happening at the same time, naturally QA and engineers tend to work together much more closely. This in turn gives considerable improvements in productivity. It nicely illustrates the phenomenon of self-organizing teams mentioned in the agile literature.
All in all transitioning to agile processes has significantly improved both our productivity as well as our accuracy, and seems very popular with our rank and file.

Traditional project management as always focused on producing various kinds of charts and reports to give management visibility and insight into the progress a project is making. The desire for this visibility off course stems from a natural skepticism that the project, left to its own devices, will get done on time. This is especially the case is with traditional waterfall projects, where development proceeds "in the dark" for long periods of time. What all of this overlooks is that schedule difficulties on projects typically stem from the individual contributors' own difficulties with correctly estimating and monitoring their own time. One of the neat things about doing agile development with a nice tool like versionone is that it not only provides management visibility into the project, it gives everyone continuous visibility into the project. Everyone can see for themselves how fast the work is going, how much is left to be done, if the project is converging. If there is 1 week left in the iteration, and people have 100 hours of work to do, then there's something wrong! Having this insight now instead of later not only allows for correction in the schedule, it gives people continuous feedback into the quality of their planning. That continuous feedback allows people to plan better. This is a lot better than a traditional scheme where you only notice at the extreme end points, that gee, this took longer than we thought it would. In particular, those developers given to hopeless overoptimism are kept in check not just by the frequent iteration milestones, but by the feedback as to how their progress/lack of progress relates to that of the rest of the team. In essence we have brought project management to the masses made everyone a project manager.

There's been much hand wringing about the loss of American software jobs to offshore developers, especially in India. Now some of this is simply people noticing that America no longer has a monopoly on computer skills, and that there is now a global market for technical skills. At the same time I think many companies that have attempted offshore development have not approached this correctly from a process standpoint. In India many things done with machines are done by hand there, simply because labor is cheap. One sees the same thing with many attempts at offshore development - there is a temptation to use a lot of cheap labor as a substitute for process. This is enforced by the fact that only part of the development lifecycle, to wit the development and/or test stages are sent offshore. With part of the work here and part of the work there, one is forced to use a strict waterfall model, with a firm spec being handed off to the other side. Such a model is known to be terribly inefficient, but it's the only way to handle this odd split. As a result much if not all of the labor gain from the offshoring is eaten up by the inefficient development model. Ironically, it many cases the result will be very unsatisfying, as the spec will, as it usually does in a naive waterfall model, turn out to be completely wrong.
Really the only way out of this dilemma is to move all of the development for something offshore, and then apply a rational agile process offshore as well. This of course means outsourcing management as well as development, and that of course rubs certain people the wrong way. But that's what makes it work. If you're not willing to do that, may be you shouldn't do the offshore thing at all. Admittedly, a major reason companies are loath to totally offshore a project is the issues around intellectual property rights. Here again, one needs to be very careful. One needs to hire people one completely trusts, and strongly bind them into the company. If one makes sure they feel like first class partners and pay them like the top people they are, then this becomes less of a danger. To the extent they are just the "cheap hired help", they won't take you so seriously either. In truth, there is no free lunch. You can hire here or you can hire there. Either way it's equally hard, and if they're far away, they're harder to oversee. If you get good people, treat them well, manage with a consistent good process here and there, then the offshore strategy can work, albeit with not nearly the savings you might first have guessed.

Anthony Bailey has written in his blog a very fine synopsis of the case for Test Driven development.
A basic premise in modern software development is pushing ownership of tasks down to the lowest level possible. What does it mean to own a piece of code ? It's more than just having some degree of control or power over it. It's about taking responsibility for that piece of code. But it isn't possible to take responsibility for something if one doesn't have reasonable grounds to believe that one can control the phenomenon in question. Unless we can ensure that the code that we own and modify actually works, and that our changes at least make it no worse, then it is foolish to take responsibility for it. So how do we ensure that the software works ?
Years ago IBM set a precedent for building highly reliable software with the software for the space shuttle's computers. Their process was simple. Each change to the system was unit tested. Once the unit tests passed, the change was integrated with the next closest layer of modules, and those were tested, and so forth, until the entire software was tested on the shuttle test bed. This method worked and still works beautifully. It requires full testing at every conceptual level of the system.
Now even if you are not building a space shuttle, you can still apply this model to normal software exercises. Notice that typical QA automation is really about integration testing of the higher levels of the system. It shouldn't even be attempted unless the system works at the lowest level. This really requires comprehensive unit tests. And the only way to ensure that you have comprehensive unit tests, and that they stay comprehensive, is to demand that the tests are built together with the code, and that every single code change be reflected in an appropriate test. Yes, in the short run it's more work. But in the long run the high quality of the resulting code will more than pay for itself in reduced rework and maintenance costs.