Latest change: 3 August 2013
Software Test Design
handbook of test analysis and design techniques
© Bogdan Bereza
my mother, who taught
To Magda, who taught me to trust
To Anna and to Robert, who still learn to tell right from wrong
To my father, who wanted to calculate
Support Team || Your
ideas are welcome || The process of writing
Now working on 2.12.17
Equivalence Partitioning (this work is not
For Whom and Why Is This Book
What Is It Needed
Testers need to know how to
design test cases. Testers' main responsibility is to
choose from infinite, or at least prohibitively huge
number of possible combinations of paths and data
values, those few - terrifyingly few - test cases that will really be exercised
during test execution and thus
be the final bulwark of defence against risk of
failure, loss and catastrophe.
What Is Available, and What is Not?
If this was not for test design, testing
would be just requirements engineering turned
upside-down, plus some knowledge in how to manage bug
reports, plus a handful of tools with funny names; a
kind of last resort fire brigade, needed only when error
prevention fails. But it is not so, because testing
means test design. Test design (and analysis, which
always precedes design) is the very essence of testing,
is what makes it a separate, special set of skills.
However, a comprehensive book
on test analysis and design techniques is missing.
There are two main reasons for this.
One reason is, testers and their employers
alike typically look for information that is needed
here and now, and which provides immediate relief.
Therefore, many books, web pages and other information
sources on testing are in one respect sadly similar to
job advertisements: they value detailed technical or
domain knowledge more than sound, generic test
knowledge. Advertisements for "test analyst with
Selenium skill and insurance industry experience" are
incomparably more common than those seeking
specialists with general test knowledge, capable and
willing to learn required domain and technical skills
fast. The result of this
is, many testers learn test design techniques on the fly
and only within given technology and domain boundaries,
and fail to see the whole picture.
I can understand, even if not completely
forgive, such approach in hasty, we-need-it-tomorrow
project situations, but the is definitely wrong approach
to learning and teaching testing. If we want really good
practitioners, we need effective textbooks for them.
These textbooks must not mix up test design skills with
everything else, including technology and domain
knowledge. Even if test practitioners need to mix these topics in project realities,
they must not mix them in their heads!
|Who cares? Well,
everybody should, because this way the
transfer of knowledge and skill is minimal.
Next time, the tester who had already learned
designing test cases based on UML activity
diagrams for a banking system, will need to
learn designing test cased based on UML state
transition diagrams for an embedded system
from scratch. This way, the waste of time and
money is maximal.
The second reason is that readers', authors'
and publishers' short-sighted motives take prevalence
over far-sighted approach when they choose to put
together test design, test organization, test management
and even test automation knowledge in one volume or on
one web site, or in one presentation. This sells better,
probably because it gives an impression of no-nonsense
pragmatism, while treating these subjects separately may
give an impression of being theoretical and academic.
Again, in order to become really high-quality
professionals, capable of applying our skills in many
different situations, we should learn - and teach -
theses areas separately. Professionals need to be able
to use their skills multi- dimensionally, not only as
one or two or even ten specific sequences.
To avoid this trap myself,
I will keep Part
1 "The Context" and Part
3 "Using Test Design to Achieve Business Goals"
of my book as short as possible, and focus
70% on Part
2 "What to Test? Test Analysis and Design Methods".
The third reason is, more than half of the
book market for testing is monopolized by ISTQB-based
textbooks. There is only a limited number of test book
copies people can buy, and since books with "ISTQB" on
the title page are safer bet for the publishers, they
dominate. Good-bye to serious test design!
And of course, the main reason is, THICK
VOLUMES SELL BAD!
|By learning test analysis
and design techniques separately, you'll be
able to use them more effectively and really
efficiently within different test strategies,
processes and test organizations, equally well
for various technologies, and in all
conceivable domains and businesses.
I searched the phrase "test
design techniques" in amazon.com, and what I got highest
up on the list was... "Exploratory Software Testing:
Tips, Tricks, Tours, and Techniques", and "Agile
Testing: A Practical Guide for Testers and Agile Teams",
and some more similar titles.
Even keeping my general reservations about exploratory
and agile approaches aside for a
while, these books were not what I was looking for: not
comprehensive textbooks on test design techniques. The
ones I found mix test design with many other issues, and
do not make any attempt to cover all available
techniques. And of course, "tips, tricks and tours" are
always easier to sell than boring lectures.
Let the search go on, back to good old names. Boris
Beizer's "Software Testing
Techniques" and "Black
Box Testing" by the same author are
very good, yet far from complete.
Glenford Myers' "The
Art of Software Testing"
(first published as early as 35
years ago!) still is and always
will be the ultimate in describing
some techniques, but definitely not
all that are worth knowing.
This Book Fills a Market Gap
of course there is Lee Copeland's "A Practitioner's Guide
to Software Test Design". Great book, in my opinion
one of the best there is on this subject. However, it is still not
I believe testers need today; it
does not offer a really comprehensive view.
Saying this, I do not mean just omitting a few
rather exotic and unimportant
techniques; I mean omitting a lot. For example, UML
contains fifteen types of models, which
useful for designing test cases;
Lee's book covers two of them use case
diagrams, and state transition
diagrams. That was the
author's choice, and I
respect it, but I
think more is needed by many testers.
There are books based
the world's most popular test
certification scheme: ISTQB Advanced
Level syllabi for test analysts and technical test
keeping my general
the contents and
structure of ISTQB
aside for a
while, the simple
ISTQB blessing present
just a smallish part
the total amount
of test design techniques. For
testing, and they
are mostly limited to
test design for
rules of static
Paul C. Jorgensen's "Software
Testing: A Craftsman's Approach"
is exceptional since it refers to
mathematical concepts and logical reasoning. At last a
book that dares use the words "discrete math" and "graph
theory"! To tell the truth, I find it hard to understand
why all other books choose to teach test design without
mathematics. Perhaps because context-driven school
pretends testing is more a social skill than anything
mathematical? However, Paul's book is very much focused
on functional testing, and formal test design: good, but
I believe non-functional testing, or experience-based
testing must not be left to intuitive magicians, but
taken care of in the same volume.
Last but not least, one must not forget Geoff Quentin's
worthwhile attempt "The Tester's Handbook", whose
chapter 7 "Specific Test techniques" contains the most comprehensive list of test techniques available now.
However, I find it not
note: I do not criticize these books, nor do
I think I am somehow wiser
then their authors. On
the contrary, I will use the knowledge they
have already provided maximally.
My wish is to
fill a market and educational gap,
not a knowledge gap.
it: my ambition
is to provide
link. Not for
the sake of theory,
nor for the
- I will
is not already
is to provide
one place the
as many as
chosen few -
book will help
testers of all
agile and traditional,
It should be
for unit-level and
or in assembly
static and dynamic,
for models and
There are some
first of all,
all of them.
helps, but the
need to make
the same kind
what to leave
book will help
them to make
Other Software Testing Knowledge Exists, and What Is
[the relationship / correlation
/ causal relationship between test design techniques
plus test coverage and failure risk (probability) || any
knowledge of intellectual and organizational
difficulties of using / deploying various test design
How to Read and Use This Book
The book is intended to be used
either as a textbook, providing as complete and
comprehensive guide as possible, or as reference book,
where each area, and each techniques, can be consulted
to some extent independently.
For the techniques described or analyzed,
most of the following issues will be covered:
1. A practical
situation where this test analysis and design
technique is helpful (including
comparative strengths & weaknesses vs.
other techniques which could be used in the
2. How much known and
widespread is the use of the technique? (a
kind of "social history" of the technique,
with links to known test syllabi and
approaches, such as exploratory testing).
description of the technique (including a way
to measure the % coverage that any given set
of test cases will obtain)
4. Practical case of
using the technique
6. Some technical and
organizational aspects of using the technique:
tools, organizational and process constraints.
Is a lightweight version of the
technique useful and used, perhaps under a
different, or even misleading name? For
example, testing invalid equivalence classes is often used
as so-called negative
example: example usage of the technique with
another technique, or some other test design
A complete reference
matrix might be tempting, but too huge for
practical purposes. To cover each pair of
technique combinations, there would be a
total number of (N-1) x (N-1) such
pairs (e.g. 29 x 29 = 841 pairs), and for
all conceivable combinations, NN
combinations (e.g 2929 =
2,6e+42)... just joking.
Only less obvious but practically promising
situations will be described. For example,
all flow graph based techniques (e.g. test
design based on use case diagrams, or
activity diagrams, or state transition
diagrams, can - and usually are - combined
with equivalence class partitioning plus
boundary value analysis, so stating this
fact many times would not be practical.
all members of the review team for your
willingness to help and support.
In order to make this
book as good and useful as possible, I
intend to make the process of writing it very transparent and interactive. The details of how the
co-operation between myself and Review Team
be performed are to be decided,
possibly influenced by - yet unknown - publisher's requirements and rules.
However, the final result of this process is to be a book, not a
blog, or a discussion forum.
It will only have
responsible author, and it will not change every day or with every tempting thought
or idea that appears now or some time in the
future. I do not think that a living
book, constantly changing, is a good idea, because focus will easily be lost,
and achieving its goal will
then be postponed for ever. Besides,
such approach may lead to the
loss of clear responsibility, or motivation,
Whether it will
be published in paper or in electronic
form, or forms, using
which model of copywright, will be decided
Avigdor M. Mevorach, Process QA Manager, Mobileye
David Hayman, Chair of ANZTB, Quality Assurance
Practice Manager Vodafone New Zealand
Declan Kavanagh, Strategic Business & IT
Derk-Jan de Grood, Holland
Edward Bishop, UK
Florian Fieber, Senior Consultant and Trainer,
Loyal Team GmbH
Ladislau Szilagyi, EuroQST, www.euroqst.ro,
Richard Taylor, Independent Consultant, UK
Tilo Linz, imbus AG, Germany
Aleksander Lipski, Poland
Dorothy Graham, software testing consultant,
speaker and author UK
Ewa Wardza│a, Independent Consultant,
Mats Wessberg, Sweden
Paul Mansell, www.thetestingrebel.com,
Piotr Kundu, Sweden
|Part 1: The Context [~10%]
Production Process and Quality Control
1.1.1 From Tribal Tools to
Software Industry: Development or Stagnation?
1.1.2 From Art and Craft to Mass Production and
1.1.3 The Deming Revolution
1.1.4 Is Software Really Special?
Chapter 1.2: History of Software
Engineering and Software Testing
1.2.1 Changing Business Context of
Software and Computers
1.2.2 From Manhattan Project to TDD
and Exploratory: the Evolution
1.2.3 Social and Political
History of Software Testing: Trends, People and
1.2.4 Is It Programming,
Computer Science or System Engineering?
1.2.5 The Rise and Fall of
Software Testing Profession
1.2.6 Is Agile Really Agile, Is
Exploratory Really Special?
Chapter 1.3: Basic
Principles of Quality Assurance and Quality Control
Principles, or Basic Misunderstandings?
1.3.2 How Do We Know It? Is This
Knowledge True, Is It Scientific?
Confusion: Project Management, Risk Management,
Testing, Requirements Engineering
1.4: Product, Project and Process Testing
What Is Product Quality?
1.4.2 Testing as Product Quality Measurement
1.4.3 Testing and Project Measurement
1.4.4 Testing as Process Measurement and
What to Test? Test Analysis and Design Methods
How Much Checking Makes Sense? Risk-based Answer
Chapter 2.2 Basic
Classification Is the Entry Point to Understanding
The Sources of Test Design Knowledge
Black and White and Many Shades
Models, Half-Models and Mental Models
Basis? What Test Basis?
Who Shall Design?
The Myth of Independent Testing
Is It Test Design, or Test Organization?
Testing and Automatic Test
2.5 How to Design Test Cases? Algorithms,
Heuristics, Intuition, Creativity, Randomness and Quackery
2.6 When to
2.6.1 As Early As Possible?
2.6.2 Depending on the Life Cycle?
2.6.3 Agile Testing, Is There Any?
2.6.4 Yes, During Test Execution as Well
2.7 Test Design for
Different Test Goals
2.7.1 Test Design for Different
2.7.1.A Principles of
Performance Test Design
2.7.2 Test Design for Various Project
2.7.1.B Principles of Usability Testing
2.7.1.C Security Testing - Is Penetration Testing
2.7.1.D There Are Hundreds of Quality Attributes - So
2.7.1.E Testing Tests: Bug Mutations
2.7.2.A Dynamic Analysis,
Regression, Confirmation, Smoke, and Acceptance Testing
- Do They Require Special Test Design?
2.7.2.B Test Design for Dynamic and Static Testing
2.7.2.C Rules of Static Analysis
2.7.2.D Designing for Different Test Levels and Test
2.8 Test Design for Various Domains
2.8.1 Testing Safety-Critical
2.8.2 How to Test Real-Time Systems?
2.8.3 Test Design Principles for Embedded Systems
2.8.3 From Here to Infinity: Business, Financial,
Medical, Insurance, Games, Web Applications...
Test Design for Various Technologies and Architectures
2.9.1 Web Testing, Mobile
Testing, Database Testing - Where Does It All End?
2.9.2 Testing Documents and Models
2.9.3 SOA, Web Services, Cloud
2.9.4 Hardware Platform: Boards, Displays, Data
2.9.5 Programming Languages
2.9.6 Operating Systems
2.10 Test Coverage and Test Design
2.10.1 Is Test Coverage a Test
Attribute, or a Test Design
2.10.2 Coverage-Based Test Design for
Static and for Dynamic Testing
2.10.3 Test Design for Functional and Structural
2.10.4 Special Case: Test Design for Model and for
Source Code Coverage
2.11 Test Design Techniques Based On Various Mathematical
and Mathematical Analysis
2.11.6 Set Theory
2.11.9 Formal Logic
Test Design Techniques for Various Model Contents
2.12.1 Class Diagrams
2.12.12 State Diagrams
2.12.13 Timing Diagrams
2.12.14 Use Case Diagrams
2.12.15 Entity Relationship Diagrams
2.12.16 Data Flow Diagrams
2.12.18 Syntax Diagrams
2.12.19 Cause-Effect Diagrams
2.12.20 Control Flow Diagrams
2.13.1 Pseudoscience -
2.13.2 Corporate Culture and Test Design
2.13.3 Group-think and Sociological Aspects of Test
2.13.4 Psychological Principles for What to Test, and
2.13.5 Special Rules - Are Bugs Really Social Creatures?
Methods To Choose?
2.14.1 Here Be Dragons! Our Knowledge Is
2.14.2 What Do Standards Say? More Anecdotes
2.14.3 How to Learn More: Connecting Test Design Techniques with
2.14.4 Bayesian Belief Nets to
Discover the Truth
|Part 3: Using Test Design to
Achieve Business Goals [~15%]
Test Case Organization and Specification
3.1.2 Conceptual versus
Executed Test Cases
3.1.3 Test Case Types
[Test conditions, test cases,
test scripts, test scenarios, test instructions, test
sequences, test matrix, test data... more?]
3.2 What Is the
Business Value of Good Testing?
3.3 Back to
Basics: Comparative Value of Various SQA Methods
3.4 How Does the
Contents of This Book Make Your Business Better?
I Know How
to Design Tests
in Many Ways,
Mow Help Me Decide What and When
- Boris Beizer "Black Box
- Boris Beizer "Software Testing Techniques"
- Cem Kaner, James Bach and
Bret Pettichord "Lessons Learned in Software Testing"
- Derk-Jan De Grood "Test Goal -
- Glenford Myers' "The Art of Software
- Geoff Quentin "The Tester's Handbook"
- Lee Copeland "A
Practitioner's Guide to Software Test Design"
- IIST Certified Software Testing
- ISTQB Foundation Level Syllabus
- "Foundations of Software Testing ISTQB
Certification" by Rex Black, Erik van Veenendaal and
- "Software Testing Foundations: A Study
Guide for the Certified Tester Exam" by Andreas
Spillner, Tilo Linz and Hans Schaefer
- "Software Testing: An ISTQB-ISEB
Foundation Guide" by Peter Morgan, Angelina Samaroo and
- "Certified Tester Foundation Level
(ISTQB) Secrets To Acing The Exam and Successful Finding
And Landing Your Next..." by Wayne Barrett
- ISTQB Advanced Level Syllabus, Test
- ISTQB Advanced Level Syllabus, Technical
- "The Software Test Engineer's Handbook:
A Study Guide for the ISTQB Test Analyst and Technical
Analyst Advanced..." by Graham Bath and Judy McKay
- "Software Testing Practice: Test
Management: A Study Guide for the Certified Tester Exam
Istqb Advanced Level" by Andreas Spillner, Tilo Linz,
Thomas Rossner and Mario Winter
- "Certified Tester Advanced Level Test
Analyst (ISTQB) Secrets To Acing The Exam and Successful
Finding And Landing..." by Janice Finch
- "Certified Tester Advanced Level
Technical Test Analyst (ISTQB) Secrets To Acing The Exam
and Successful Finding..." by Russell Price
- "ISTQB Advanced Functional Exam
Preparation Guide" by Rex Black
- "Advanced Software Testing - Vol. 1:
Guide to the ISTQB Advanced Certification as an Advanced
Test Analyst" by Rex Black
- "Advanced Software Testing - Vol. 3:
Guide to the ISTQB Advanced Certification as an Advanced
Technical Test Analyst" by Rex Black and Jamie L
- "Guide to Advanced Software Testing" by
Anne Mette Jonassen Hass
- QAI CAST Syllabus (Certified Associate in
- QAI CSTE Syllabus (Certified Software
- Torbj÷rn Ryber "Essential Test Design"