Thinking about Quality

All competent companies engage in testing of their software. When this testing is completed, and the software meets the agreed upon quality criteria, the software is viewed as being good enough to release to customers. Let's dissect this further. Typically a quality plan specifies a basic set of regression tests describing the base functionality of a product, a set of new test cases for new features, and certain generic cases covering reliability, performance and scalability. In essence we have an enumerated set of behaviors, and if the software matches those behaviors, then we say it is of good quality and we release it.
And then what happens ? Will the customer also think this a high quality product ? Obviously the customer will appreciate the product's quality only if the customer's exact use cases are exactly covered in the test cases run back home. Ideally, no matter what the customer tries, everything works exactly as it should. Then the customer will no doubt be well pleased. In real life of course, some customers are never happy, and few customers are always happy about everything.
The question is, what is the perceived quality of the product, and how does map to the formal quality measured in engineering ? How can we quantify the perceived quality of the product ? If we take steps to improve the formal quality of the product, how can we measure the improvement in perceived quality ?
One obvious dimension of perceived quality is reliability. That is, the set of customers perceives each instance of a defect as an atomic thing. So even if there is only one defect, if it occurs every 15 minutes, it will be perceived as many, many incidents. In that sense perceived quality is influenced by reliability, that is, the distance between occurrences of defects.
A subtler question is, who are the customers ? For an enterprise software product, in many cases the actual end users hardly have a voice; it is the perceptions of the executive sponsor that matter. In a call center, quality is more about what metrics of call center efficiency show, than about what the agents think. The answer to who is the customer that perceives quality may also change over time. While the software is being sold, its demo quality as perceived by sales people and prospects matter. When it's being implemented, partners and professional services determine quality. When the software is in production, sometimes end users, and sometimes their managers determine the perceptions.
This means that we need to formally study the reliability of the system as it applies to different classes of users at different phases in the life cycle of an implementation in order to characterize the perceived quality of the system.
We'll dig deeper into this next time.

New comers to agile really wonder about the wisdom of the daily meetings. Why have yet another meeting ? Can't one do it by email, or maybe once a week ? Well you could, but you shouldn't. The daily meeting is very different from other meetings in that while most meetings are called to solve problems, the daily meeting serves to prevent problems. It is daily, because this is a quick way of resolving problems as they come up, and not waiting all week for them to get better. It's better than email, because it encourages face to face communication. In fact in general we have been pushing people to meet face to face all the time rather than sending emails or even worse reassigning requests from person to person in a tracking system. The idea is that personal contact is always more efficient communication than impersonal contact. Understand each other, and solve the problem, together, NOW.
In fact, the daily meeting serves a couple of purposes. It allows us to quickly resolve current issues. It allows us to collectively plan our work for the next day. It does these things so efficiently, that the other meetings we used to have about status have evaporated. So it turns out the meeting adds no extra time at all! But beyond those tactical things, it also brings us together as a group, allows us to synch up our attitudes and feelings, and improve team work. It gives us a sense of camaraderie that's really important.

This is a really interesting article about the problems involved in testing large systems. It make a couple of good points:
- Code Coverage measurement alone is an insufficient criterion of test effectiveness without understanding the quality of result validation.
- Effective unit testing is the only way to make overall systems testing tractable.
- Model based testing has a lot of appeal at the system test level.
- Static analysis is a powerful adjunct to unit testing.
- Code Coverage measurement is a good way of prioritizing tests.
- Dependency analysis is a good way of pruning test sets.
- All pairs is a good way of managing large platform matrixes.
Lots of food for thought there!