Thinking about Quality

As we've started using VersionOne to track our progress a number of interesting things happened. Originally we were only going to have the managers using it. But suddenly everyone wanted an account. Now everyone is using it. Wonder of wonders! In the past, when we used MS Project, people treated schedules and the like as if they had to have another spoon of cod liver oil. But VersionOne has the interesting effect of making project management fun. Since people enjoy using the tool, they do use it, and start entering enough information. In general the bane of most metrics projects is that there is a shortage of good source data because most people only fill it in a very cursory way, just enough to get by. But now that we have an intrinsically appealing tool, we are getting lots of data. This in turn makes the progress of our world a lot more visible. All the little dashboards, graphs and charts in VersionOne are interesting not just because they look cool, but because there's enough data behind them so that you can tell what's going on. This really underscores the importance of good user interface design in development tools of any kind, because it can really affect the acceptance and effectiveness of the tools with the developer community.

I started a small experiment to try VersionOne as an alternative to MS Project. Since then the small experiment has turned into a forrest fire. Everyone wants to use it. It is truly an amazing tool. What's so neat about it is its economy, it's generality and it's ease of use. It has a very straightforward and flexible data model. The UI very transparently exposes this data model, and allows for very straightforward navigation. It requires precious little documentation, and can be easily figured out in no time. It has lots of neat reports and displays that both measure progress and point out errors in the data entry. This last point is particularly helpful, because on any nontrivial project keeping the data scrubbed is always a major issue.
The data model is straightforward. You create a set of members, admin or regular. Admin's can create users. You then create products (= aka projects), and assign members to them. Within the project you define releases, iterations and teams. Members are assigned to teams. Releases and iterations enjoy a many-many relationship. You can then create themes and stories. Stories are assigned to themes, iterations and releases. Finally you build tasks that are assigned to stories. That's it practically! But you can accomplish a lot with that. All objects in the system are owned by someone which facilitates hunting down who's working on it.
You can customize it for different agile methodologies. But no matter which one you're using, it's a joy to use because it's so simple, elegant and general. This is definitely one of the nicest tools I've found in a long time.

Let's say we are building a system using a scrum like process and we have n-iterations of development, capped by a final iteration focusing just on QA and bug fixing. QA is entirely active throughout this process. How does the work of QA differ between the earlier iterations and the last one ? The final iteration corresponds well to the traditional QA phase in a waterfall process in that presumes the system is complete, the test plans are ready, and then it proceeds by assuming that nothing about the system works, and systematically checks every aspect of the system. The earlier phases though have a different character. As a start, QA needs to build test plans for new features as they appear. Then, automation has to be built and modified to cope with the new features. Most importantly, there is a continuous process of regression testing to verify that the system is working. Here there is something different though. To the extent that the testing is not automated, and we always will have a bunch of exploratory manual testing, we do not proceed from the assumption that nothing works. We can't do that because it's too much work to test everything. Indeed we don't have to. Because the system is working at the start of the first iteration, and we demand that the system be working at the end of each iteration, we can safely assume that most of the system works. As a result, we can focus our testing on those areas which we believe are likely to break. Since we won't always be able to guess precisely what those areas are, we can also create some useful realistic end-to-end user scenarios that exercise wide swaths of the system. In this way we identify strategic subsets of tests that we run over and over again to see what doesn't work. So in the early iterations we divide actual testing between testing new features, testing risky areas, and trying out key user scenarios. Only at the end do we go back to the traditional mode of doing a comprehensive quality audit. Obviously, to the extent that we can fully automate the testing, we can reduce any exposures here. But large, complex system may often have significant areas that still warrant manual testing, and the heuristics described here can prove useful for simplifying what needs to be tested.