Thinking about Quality

How far do we go in specifying processes ? Suppose we allow teams to self-organize within certain parameters. How far should we go in specifying their processes ? Where is the boundary between our standards and their freedom to self-organize ? For example in a large organization with many teams, do we impose the same methodology on all teams ? An approach is to say that we strictly specify everything that is essential to the teams working together productively and leave the rest up to team choices. It's much the same principle as in a democracy: democracy doesn't imply freedom from laws, it implies that freedom is enabled and protected by laws. That's why democracy is not mob rule. One needs to have enough rules to allow people to peacefully coexist and be able to exercise their freedoms. The same principle should be translatable to organizations. We carefully specify what we need to in order to ensure that the teams are able to make progress achieving the company's goals. But we intentionally leave unspecified everything beyond those minimal constraints. Still where should we draw the line ? Ultimately the teams should meet together and reach a group consensus as to where the line should be drawn. Once everyone agrees to a set of rules, then everyone is free to enjoy playing the game according to those rules. From this we get two interesting results: that the standards are themselves the product of self-organization, and that self-organization in no way implies that we don't have fully specified processes, instead that the processes are fully specified to some precise and important domain boundary that is quite intrinsic to the process. This is why we don't give up CMM rigor even in an agile world: it's simply important to be clear on the exact context in which we create fully specified processes. It's much like in the chemistry of gases, we compute exactly on a volume of gas, without worrying about the motions of particular molecules within the gas.

An interesting question in thinking about the makeup of software teams concerns performance management and the relative strength of a team. Many studies have shown that in software the best performers are vastly more productive than more average performers. Of course, this means that at least in theory one would want teams of only very high performers. Is this what one should strive for? Certainly as an abstract goal, trying to hire the very best people is impossible to argue against. Of course it's a good thing to do. The question is: are you consistently able to find those top performers? Having found them can you retain them ? Will others grow into that role ? Will some over time lose that great productivity ? My own empirical observation is that for most companies finding and keeping the very best talent is at best a hit or miss affair. The teams will have some high performers and a lot of more average ones, and even worse, that management is fairly poor at correctly classifying who is who. It's also the case that having a great software team is NOT correlated with business success. Many bankrupt companies had quite good software teams; there are quite a few highly profitable enterprises with fairly mediocre teams. How can this be ?
It all gets back to the issue of prioritization. Most companies fail to prioritize properly and accordingly build too much software. Thus having a great team is wasted, because the great team builds things that are strictly speaking unneeded. In other organizations, even with lower productivity, a clear sense of what actually needs to get done allows the teams to be successful.
If a company has a very precise understanding of its goals, a well organized team effort can more than make up for individual weaknesses. The methodical team doing middle of the road things in support of business goals is less romantic than the heroic cowboy programmer who fueled only by pizza and pepsi cranks out reams of code day after day without sleeping once, but in many cases is just what is required to achieve business goals.
Furthermore, many modern practices such as pair programming, and extensive reviews and inspections allow a piece of software to be built by a group of people working together rather than by a single individual. In this case the overall ability of the team to collaborate effectively may be even more important than the skills of any one team member. In this respect we may need to reassess how we think about the skills of employees. It may matter as much how well they collaborate on a team as they do working individually.

What sets humans apart both from other animals as well as machines is that not only do humans bring great intelligence to bear on a problem, they also have the capacity for hope. Hope is an irrational thing; it leads people to strive to do the improbable or seemingly impossible. We all know of people who persevered against apparently impossible odds. If we want to create a system of the highest quality, we need certainly need good processes, because good processes enable quality. But what actualizes quality is the desire of the team to produce good stuff. There is this extra bit of motivation that causes the ok to turn into the excellent. It can come from many sources, the promise of riches and success, simple group spirit, enthusiasm for the act of creation. This spirit is easily wrecked, however. All it takes for expectations to be dashed, great effort to be wasted, ineptitude to be rewarded. One of the big challenges in management is to nurture the spirit that causes high quality, in spite of the ups and downs of business and of ordinary human failings.
A major component is having a good quality team itself, because only a group of compatible good people can have this spirit. Another component is having a good environment that notices and rewards good work. Another component is a shared vision, the common hope that things will bet better.
While these factors influence just the work environment, once the team has the right spirit, the right results will ensue. Good processes help by gathering and channeling this positive energy. This is the duality of the rational and the irrational in the quality of organizations.

An important part of management is appraising performance. This has a couple of aspects. In one sense, one needs to be able to give individuals guidance on their career, recognizing achievements, encouraging growth, and helping with problems. In a broader sense, performance appraisal helps one assess the strength of one's team, to determine both strengths and weaknesses. Since their is a such a gap between the top performance and everyone else, one obviously wants to further the top performers. At the same time, the bottom end must be removed, because low performers hurt morale by their mere presence, and often amount to an actually negative contribution to team productivity due to all the management attention they suck up. Because of this, as a manager it is important to classify workers to identify both the stars as well as the people in trouble.
Of course, there's more. Inevitably HR departments want to assign some rating to people. The question is, what rating do you assign to the people in the middle ? Are they truly mediocre, taking that last phrase literally ? If one has done a good job recruiting, I don't think they are mediocre at all. In a good team, you'll have lots of solid, hard working contributors. A few of them will rise above the fray as true stars, but it's the other folks who still carry the bulk of the load. And these people may be in the middle of our distribution, but they are still really good employees and should be recognized as such! Any grading system that says that they are only "medium" is an insult to their dedication and contribution. My own view is that I don't really want anyone who is truly mediocre on my team. Since I tend to weed out anyone on the bottom of my distribution, the folks who remain are really all quite good. I want to say there are "great" people, there are "good" people, and there are people who will soon no longer work for me. This is quite important. I want nothing less than good people working for me, and to the extent they are good, I want them recognized as such.

A great frustration to many in planning releases is that one just doesn't have enough information. Requirements are incomplete, technology is unknown, future distractions are unpredictable, and one just hasn't figured out yet how to do certain things. Even though you know you're supposed to write "new reports" for some feature, you really have no idea what that means. Under these circumstances how can one agree to a set feature list, or even worse, a set feature list on a set schedule? You can't of course.
The trick to success is to give up figuring it all out. Just figure out whatever you do know about, and make sure you carefully evaluate the priority, risk and dependencies of the pieces you know about. Construct a plan that's detailed about what you understand, and then gets fuzzier about the pieces that are fuzzier. Then organize your first iteration around the best known pieces. By the time you finish that iteration, you'll probably know more about some of the other pieces, and you can reprioritize. The agile approach really saves the day here, because it lets you work with partially understood information.
In switching to agile then one of the hardest parts is "giving up". You have to give up trying to know it all, and accept your partial ignorance of the situation. Key is the word partial. If you're completely ignorant you're still in trouble :) . But as long as you know something, then you can get started. Of course, it does help as a thought experiment to try to work out enough of the whole to convince yourself that in general the plan is plausible. You don't want to get 80% of the way in only to find out that success is in fact conceptually impossible. This is especially true in situations with complex external interfaces or technical risks. But don't confuse this thought experiment with a binding schedule. What matters is a continuously updated fully prioritized plan, that reflects you're continually evolving best understanding.

One of the hardest parts of product development is release planning. It's hard because for engineers it's much easier to figure out how to do something than it is to figure out what to do. In fact this is hard for anyone. Most people just hope someone else will do it. So the team gets together to figure out what general feature set to include in a release. It's natural to track these features with a spreadsheet. On a big product built over many releases, these spreadsheets become very long. Now they are useful to capture the long lists of stuff. But they're bad news for planning a new release, because they concentrate the team on details. When planning a new release, you should be thinking about the big picture: how will this release help the business ? What can we do that makes a difference ? That anyone will notice ? At this point a long list will simply distract. You want to make a list of major themes and prioritize them carefully. Once you've agreed on a prioritized list of themes, then you can drill down and work on more detailed lists, but not before. Even a large system shouldn't have more than 12 or so themes at the highest level to prioritize between. This also helps guard against the Microsoftish tendency to simply pile features on top of one another. The best application is not the one with the most features or checkoffs. The best application is the one that delivers the most real life value to the customer while ensuring a reasonable profit to it's maker. And you have to be a zealot about focusing on those things and only those things that contribute to that.

There are few arguments as much fun as trying to figure out what is a regression, why there are so many of them, and how to prevent them. We define a regression as:

Any functionality that worked in a previous build of the current development version or in any build of a previous release and does not work now is a regression. The only exception to this definition is the case where the test case is not valid in the current release.

We like to track which defects in our defect database are regressions. There's often a good bit of debate about it. Basically, developers often feel that a regression is where code that used to work doesn't work any more, whereas QA feels that functionality that used to work but doesn't anymore is a regression. QA in this case is closer to the customer perspective.

That said, what to do about it ? Developers want to get at the causes of regressions, because frequently small code changes in a key module can causes widespread failures on the surface. In this sense the quantity of regressions is not a fair assessment of how much underlying breakage there is. Looking then at the causes, analysis shows that in most cases most regressions are caused by adding new features. Duuuuuuh. In other words, changing a system causes regressions. This isn't very helpful. Actually, from a quality perspective, it's of less interest why it broke, then that it broke. For testing, we really want to focus not on the areas that cause regressions, but on the areas that manifest regressions. This is what the customer will notice, and it's where you'll find regressions. You can't fix it until you've found it. This point is especially important when adding automated tests to existing systems, because there is much less of a need to achieve complete coverage, than to identify the "canary in the coal mine" spots that tend to break often and are often symptomatic of deeper issues. When applied to QA automation, this also helps deal with the common problem that unit tests aren't sophisticated enough to cover the actual failure conditions, and one needs some well chosen end-to-end automation to really check system health.

Agile methods tend to emphasize "people" over "processes and tools". What's meant by that? A couple of things. First, it emphasizes that processes should follow the way people work, rather than making people do things they wouldn't otherwise do according to some process. This isn't good enough though, because any process becomes vacuous if people don't follow it. The deeper meaning is that if one has a good set of people one can trust them to do the work, and in that case, giving them a natural process is optimal because one is trusting them to work well. This emphasizes a dichotomy in processes. If one has a set of mediocre people one cannot intrinsically trust, then one can use very heavy handed processes to make sure they produce some consistent result. But in our present case if we make sure we have really excellent people, then we don't have to second guess them as much, and we can get away with a lot less process.
But how far to trust ? It is very important to have some external validation of all work. Even in a world with minimal processes, we need checks and balances. A simple standard is that no task is done unless someone besides the person doing the work validates that it is done. That is, a piece of code should have been reviewed, tested, demonstrated to some other person or persons, and a document should have been reviewed by someone else. That external validation keeps everyone honest. The gotcha is that the whole system breaks down if routinely tasks cannot be successfully validated. In a control oriented regimen heavier processes are instituted to catch such failures earlier. In the agile approach a greater emphasis is put on having good people so that tasks do get done well. This places a great responsibility on management to really aggressively manage the performance of the staff, and make sure all the staff is worthy of this level of trust.

I'm currently evaluating VersionOne, which is a really nice project-tracking tool for agile teams. What's neat about is that it has a nice simple but powerful structure. It's quite general, allowing a hierarchy of products that have releases. The releases have stories aka features. The releases are divided into iterations, to which are assigned the stories. Each story can have multiple tasks and tests. It's all quite simple really. Everything is nicely interconnected, so navigation is very natural. It even works with Firefox, even though it's a .NET app. It also scales ok if you put lots of data. Everything can be imported from or exported to Excel. It's got nice reporting and graphs to track progress. It's really quite a nice system! It seems a lot simpler and friendlier than horrible long Project files. We'll give this a try and see how it goes. Anything has to be better than MS Project. ugggh.......Changing the subject, I just thought of a nice encapsulation of our latest round of process improvement. Our methodology should reflect how we work, rather than our work reflecting our methodology.

Grady Booch has embarked on a very ambitious project to create a Handbook of Software Architecture that would rival Donald Knuth's effort, except cataloging the fundamental patterns of software rather than algorithms. He's especially interested in the higher level patterns that occur in nontrivial systems. You have to register to see all of it (it's work in progress), but the plan seems fascinating. There are a couple of main parts to the work: a general overview of architecture, an inventory of major software systems and their attributes, and then an analysis of the patterns found in these systems. We'll have to keep watching this one!

Question: How can we use earned value to track incremental development ?
Let's suppose we plan a release as containing some set of features. We'll estimate how much effort each feature takes, roughly. Adding up the estimates tells us how much earned value we need to produce. If we assume constant staffing, then according to plan earned value will grow linearly with time. If staffing is not constant, and we reflect our staffing plan, then earned value proceeds nonlinearly with time.
So we start the project. At the end of each iteration, we can calculate how many units of earned value we have actually produced. We'll have learned more about how much work is really left to be done, and we can calculate a new amount of work left to do. The amount of work we've actually done is graphed against time. We use that velocity together with the new estimate of total work to tell when we'll be done. Obviously, if the done date is further in the future then we'd wish, then we need to either cut features, or figure out some way of increasing the rate of production.

I've made a list of good books on Agile Development Methodologies, which complements my existing list of Classics of Software Engineering.

You may find these useful in your intellectual travels!

Very large multi-team projects cannot, as a totality, be run with something as simple as a pure XP model. Then again, a pure waterfall model doesn't work very well either. One transitional approach is to adopt an agile project management and process model, and then allow individual teams to move at their own pace. Here's how this can work:
We divide the release into a series of chunks of identical length. At the end of each chunk, all work done in that chunk should be completed, QA'd and demoable. After a couple of such chunks, we devote one chunk to nothing but QA and bug fixes, and out the door it goes! Within each chunk we rely on continuous regression testing, primarily automated. These tests must pass at least once a week, which implies almost continual attention to failures and blocking issues. When we do our project plan we carefully prioritize our committed items, putting all the highest risk and/or highest priority items in the first few chunks, and the lowest priority items in the last few chunks. We also identify dependencies, and distribute work items so that dependent items come after their dependencies. Near but before the end of each chunk, we replan the schedule to decide exactly what goes into the next chunk, so that at the appointed time everyone has work assigned. Note that the chunks always finish on time, only their content can change.
Overall we can track progress via an earned value approach, where we know how many items per chunk need to get built to produce the product, and we track our actual velocity at getting them done. This helps tell us if we're working fast enough to get the job done, or if there are risk factors. Adding a notion of available staffing helps measure both efficiency and distraction factors. Also key to this approach is just-in-time scheduling, design, test design. We really only build what we need, when we need. However, this is monitored by our ongoing prioritization and risk analysis process, which reprioritizes the list of tasks as we move from chunk to chunk.
In this model QA spends its time on the one hand building automation, on the other hand doing test planning, automated regression testing, and manual exploratory testing.
Each team is free to organize it's specific scheduling and coding as it sees fit. Some teams may try a pure XP approach, others a more conventional mini-waterfall approach. There is flexibility here. It is this flexibility that allows for gradual cultural change towards more flexible models.

People over Processes is a basic agile tenet. Today we found a good application. We've been holding for years daily meetings during the endgame of releases to triage bugs. We did that because in the past we had inept managers and we needed the meeting to ensure quality decisions. We noticed that these days the managers on their own correctly triage the bugs that come in the morning. We've been only using the afternoon meeting for the afternoon bugs. This is silly. Trust the managers to do their work all the time, and take note if they start making bad decisions. And cancel the meeting. This is what "People over Process" means. It doesn't mean not having a process. It does mean having processes that imply trust in people and are correspondingly much more lightweight than a process that specifically doesn't trust people. It also doesn't mean that we lack check and balances, because we can use our reports to check if bad decisions are made, but it does mean we don't try to use the process a substitute for intelligent realtime decision making.