Tuesday, May 13, 2008

Test Coverage, some rules of thumb

Summary: Quick rules to keep in mind when thinking about test coverage

As a tester, are you ever asked by a stakeholder to describe what tests you have and how they cover the application? Here are some rules-of-thumb to think about before you get the questions.

Find out what matters:
Start by asking questions of the stakeholders. Listen, listen, listen to their answers.

Report on coverage in terms of what matters: The CIO may not care about unit-level coverage metrics (lines of code, classes, etc...) She probably will care about whether key business processes supported by the system have been tested. Your report to this audience should probably orient coverage towards business processes.

There may be different view of what matters: I mentioned things the CIO may care about, but the network security organization will probably care greatly about access control and other types of security testing. Can your coverage reporting mechanism accommodate different views of coverage?

What matters changes: The things that are important at the outset of a development effort may not stay important. Things that are unknown at the outset may wind up being very important. How easy is it to modify information about what tests cover?

Talk about what matters at the highest (useful) level: If we're doing things c
orrectly, we're probably tracking coverage information at very fine-grained levels. There is a temptation to show all the details up front. Resist this....not because you don't want to share the details, but because details can confuse the bigger picture. A picture like this should not start discussions about coverage.

This is a better choice:

With that said, you want to be able to drill down easily into greater detail.

Future posts will dive into the nitty-gritty of following these rules-of-thumb.


volcker said...

Isn't that the eternal challenge, distilling the essential...


Cliff said...

You're absolutely right, Ed. It is an eternal challenge. This point was buried in my post somewhat, but it's important to remember that whats essential changes or at least appears to (actually I think it's fairly constant, but just becomes more easy to spot over time). The takeaway? That you don't have to get it perfect upfront (and that your test coverage reporting mechanism should make it less painful to adjust things when you find you need to). thanks for the comment :)