Model-based testing
Encyclopedia
Model-based testing is the application of Model based design
for designing and optionally executing the necessary artifacts to perform software testing
. Models can be used to represent the desired behavior of the System Under Test (SUT), or to represent the desired testing strategies and testing environment. The picture on the right depicts the former approach.
A model describing the System Under Test (SUT) is usually an abstract, partial presentation of the system under test's desired behavior.
The test cases derived from this model are functional tests on the same level of abstraction as the model. These test cases are collectively known as the abstract test suite. The abstract test suite cannot be directly executed against the system under test because it is on the wrong level of abstraction. Therefore an executable test suite must be derived from the abstract test suite that can communicate with the system under test. This is done by mapping the abstract test cases to
concrete test cases suitable for execution. In some model-based testing tools, the model contains enough information to generate an executable test suite from it. In the case of online testing (see below), the abstract test suite exists only as a concept but not as an explicit artifact.
There are many different ways to "derive" tests from a model. Because testing is usually experimental and based on heuristics, there is no one best way to do this. It is common to consolidate all test derivation related design decisions into a package that is often known as "test requirements", "test purpose" or even "use case". This package can contain e.g. information about the part of the model that should be the focus for testing, or about the conditions where it is correct to stop testing (test stopping criteria).
Because test suites are derived from models and not from source code, model-based testing is usually seen as one form of black-box testing. In some aspects, this is not completely accurate. Model-based testing can be combined with source-code level test coverage measurement, and functional models can be based on existing source code in the first place.
Model-based testing for complex software systems is still an evolving field.
Especially in Model Driven Engineering or in OMG's model-driven architecture
the model is built before or parallel to the development process of the system under test. The model can also be constructed from the completed system.
Recently the model is created mostly manually, but there are also attempts to create the model automatically, for
instance out of the source code. One important way to create new models is by model transformation
, using languages like ATL
, a QVT
-like Domain Specific Language.
Model-based testing inherits the complexity of the domain or, more particularly, of the related domain models.
Online testing means that a model-based testing tool connects “directly” to a system under test and tests it dynamically.
Offline generation of executable tests means that a model-based testing tool generates test cases as a computer-readable asset that can be later deployed automatically. This asset can be, for instance, a collection of Python
classes that embodies the generated testing logic.
Offline generation of manually deployable tests means that a model-based testing tool generates test cases as a human-readable asset that can be later deployed manually. This asset can be, for instance, a PDF document in English that describes
the generated test steps.
Often the model is translated to or interpreted as a finite state automaton or a state transition system
. This automaton represents the possible configurations of the system under test. To find test cases, the automaton is searched for executable paths. A possible execution path can serve as a test case. This method works if the model is deterministic or can be transformed into a deterministic one. Valuable off-nominal test cases may be obtained by leveraging un-specified transitions in these models.
Depending on the complexity of the system under test and the corresponding model the number of paths can be very large,
because of the huge amount of possible configurations of the system. For finding appropriate test cases, i.e. paths that refer to a certain requirement to proof, the search of the paths has to be guided. For test case generation, multiple techniques have been applied and are surveyed in.
.
can be used to select test cases satisfying specific constraints by solving a set of constraints over a set of variables. The system is described by the means of constraints. Solving the set of constraints can be done by Boolean solvers (e.g. SAT-solvers based on the Boolean satisfiability problem
) or by numerical analysis
, like the Gaussian elimination
. A solution found by solving the set of constraints formulas can serve as a test cases for the corresponding system.
Constraint programming can be combined with symbolic execution. In this approach a system model is executed symbolically, i.e. collecting data constraints over different control paths, and then using the constraint programming method for solving the constraints and producing test cases.
can also be used for test case generation. Originally model checking was developed as a technique to check if a property of a specification is valid in a model. When used for testing, a model of the system under test, and a property to test is provided to the model checker. Within the procedure of proofing, if this property is valid in the model, the model checker detects witnesses and counterexamples. A witness is a path, where the property is satisfied, whereas a counterexample is a path in the execution of the model, where the property is violated. These paths can again be used as test cases.
specifically, a GUI is decomposed into a hierarchy of modal dialogs; this hierarchy is represented as an integration tree; each modal dialog is represented as an event-flow graph that shows all possible event execution paths in the dialog; individual events are represented using their preconditions and effects. An overview of the event-flow model with associated algorithms to semi-automatically reverse engineer the model from an executing GUI software is presented in this 2007 paper. Because the event-flow model is not tied to a specific aspect of the GUI testing process, it may be used to perform a wide variety of testing tasks by defining specialized model-based techniques called event-space exploration strategies (ESES). These ESES use the event-flow model in a number of ways to develop an end-to-end GUI testing process, namely by checking the model, test-case generation, and test oracle creation. Please see the GUI Testing page for more details.
(FSM) which represents all possible usage scenario of the system and the Operational Profiles (OP) which qualify the FSM to represent how the system is or will be used statistically. The first (FSM) helps to know what can be or has been tested and the second (OP) helps to derive operational test cases.
Usage/Statistical Model-based Testing starts from the facts that is not not possible to exhaustively test a system and that failure can appear with a very low rate. This approach offers a pragmatic way to statically derive test cases focused on: improving as prompt as possible the system under test reliability.
Model based design
Model-Based Design is a mathematical and visual method of addressing problems associated with designing complex control, signal processing and communication systems. It is used in many motion control, industrial equipment, aerospace, and automotive applications...
for designing and optionally executing the necessary artifacts to perform software testing
Software testing
Software testing is an investigation conducted to provide stakeholders with information about the quality of the product or service under test. Software testing can also provide an objective, independent view of the software to allow the business to appreciate and understand the risks of software...
. Models can be used to represent the desired behavior of the System Under Test (SUT), or to represent the desired testing strategies and testing environment. The picture on the right depicts the former approach.
The test cases derived from this model are functional tests on the same level of abstraction as the model. These test cases are collectively known as the abstract test suite. The abstract test suite cannot be directly executed against the system under test because it is on the wrong level of abstraction. Therefore an executable test suite must be derived from the abstract test suite that can communicate with the system under test. This is done by mapping the abstract test cases to
concrete test cases suitable for execution. In some model-based testing tools, the model contains enough information to generate an executable test suite from it. In the case of online testing (see below), the abstract test suite exists only as a concept but not as an explicit artifact.
There are many different ways to "derive" tests from a model. Because testing is usually experimental and based on heuristics, there is no one best way to do this. It is common to consolidate all test derivation related design decisions into a package that is often known as "test requirements", "test purpose" or even "use case". This package can contain e.g. information about the part of the model that should be the focus for testing, or about the conditions where it is correct to stop testing (test stopping criteria).
Because test suites are derived from models and not from source code, model-based testing is usually seen as one form of black-box testing. In some aspects, this is not completely accurate. Model-based testing can be combined with source-code level test coverage measurement, and functional models can be based on existing source code in the first place.
Model-based testing for complex software systems is still an evolving field.
Models
Model-driven architecture
Model-driven architecture is a software design approach for the development of software systems. It provides a set of guidelines for the structuring of specifications, which are expressed as models. Model-driven architecture is a kind of domain engineering, and supports model-driven engineering of...
the model is built before or parallel to the development process of the system under test. The model can also be constructed from the completed system.
Recently the model is created mostly manually, but there are also attempts to create the model automatically, for
instance out of the source code. One important way to create new models is by model transformation
Model transformation
A model transformation, in model-driven engineering, is an automatable way of ensuring that a family of models is consistent, in a precise sense which the software engineer can define...
, using languages like ATL
ATLAS Transformation Language
ATL is a model transformation language and toolkit developed and maintained by OBEO and AtlanMod. It was initiated by the team...
, a QVT
QVT
QVT is a standard set of languages for model transformation defined by the Object Management Group .- Overview :...
-like Domain Specific Language.
Model-based testing inherits the complexity of the domain or, more particularly, of the related domain models.
Deploying model-based testing
There are various known ways to deploy model-based testing, which include online testing, offline generation of executable tests, and offline generation of manually deployable tests.Online testing means that a model-based testing tool connects “directly” to a system under test and tests it dynamically.
Offline generation of executable tests means that a model-based testing tool generates test cases as a computer-readable asset that can be later deployed automatically. This asset can be, for instance, a collection of Python
Python (programming language)
Python is a general-purpose, high-level programming language whose design philosophy emphasizes code readability. Python claims to "[combine] remarkable power with very clear syntax", and its standard library is large and comprehensive...
classes that embodies the generated testing logic.
Offline generation of manually deployable tests means that a model-based testing tool generates test cases as a human-readable asset that can be later deployed manually. This asset can be, for instance, a PDF document in English that describes
the generated test steps.
Deriving tests algorithmically
The effectiveness of model-based testing is primarily due to the potential for automation it offers. If the model is machine-readable and formal to the extent that it has a well-defined behavioral interpretation, test cases can in principle be derived mechanically.Often the model is translated to or interpreted as a finite state automaton or a state transition system
State transition system
In theoretical computer science, a state transition system is an abstract machine used in the study of computation. The machine consists of a set of states and transitions between states, which may be labeled with labels chosen from a set; the same label may appear on more than one transition...
. This automaton represents the possible configurations of the system under test. To find test cases, the automaton is searched for executable paths. A possible execution path can serve as a test case. This method works if the model is deterministic or can be transformed into a deterministic one. Valuable off-nominal test cases may be obtained by leveraging un-specified transitions in these models.
Depending on the complexity of the system under test and the corresponding model the number of paths can be very large,
because of the huge amount of possible configurations of the system. For finding appropriate test cases, i.e. paths that refer to a certain requirement to proof, the search of the paths has to be guided. For test case generation, multiple techniques have been applied and are surveyed in.
Test case generation by theorem proving
Theorem proving has been originally used for automated proving of logical formulas. For model-based testing approaches the system is modeled by a set of logical expressions (predicates) specifying the system's behavior. For selecting test cases the model is partitioned into equivalence classes over the valid interpretation of the set of the logical expressions describing the system under development. Each class is representing a certain system behavior and can therefore serve as a test case. The simplest partitioning is done by the disjunctive normal form approach. The logical expressions describing the system's behavior are transformed into the disjunctive normal formDisjunctive normal form
In boolean logic, a disjunctive normal form is a standardization of a logical formula which is a disjunction of conjunctive clauses. As a normal form, it is useful in automated theorem proving. A logical formula is considered to be in DNF if and only if it is a disjunction of one or more...
.
Test case generation by constraint logic programming and symbolic execution
Constraint programmingConstraint programming
Constraint programming is a programming paradigm wherein relations between variables are stated in the form of constraints. Constraints differ from the common primitives of imperative programming languages in that they do not specify a step or sequence of steps to execute, but rather the properties...
can be used to select test cases satisfying specific constraints by solving a set of constraints over a set of variables. The system is described by the means of constraints. Solving the set of constraints can be done by Boolean solvers (e.g. SAT-solvers based on the Boolean satisfiability problem
Boolean satisfiability problem
In computer science, satisfiability is the problem of determining if the variables of a given Boolean formula can be assigned in such a way as to make the formula evaluate to TRUE...
) or by numerical analysis
Numerical analysis
Numerical analysis is the study of algorithms that use numerical approximation for the problems of mathematical analysis ....
, like the Gaussian elimination
Gaussian elimination
In linear algebra, Gaussian elimination is an algorithm for solving systems of linear equations. It can also be used to find the rank of a matrix, to calculate the determinant of a matrix, and to calculate the inverse of an invertible square matrix...
. A solution found by solving the set of constraints formulas can serve as a test cases for the corresponding system.
Constraint programming can be combined with symbolic execution. In this approach a system model is executed symbolically, i.e. collecting data constraints over different control paths, and then using the constraint programming method for solving the constraints and producing test cases.
Test case generation by model checking
Model checkersModel checking
In computer science, model checking refers to the following problem:Given a model of a system, test automatically whether this model meets a given specification....
can also be used for test case generation. Originally model checking was developed as a technique to check if a property of a specification is valid in a model. When used for testing, a model of the system under test, and a property to test is provided to the model checker. Within the procedure of proofing, if this property is valid in the model, the model checker detects witnesses and counterexamples. A witness is a path, where the property is satisfied, whereas a counterexample is a path in the execution of the model, where the property is violated. These paths can again be used as test cases.
Test case generation by using an event-flow model
A popular model that has recently been used extensively for testing software with a graphical user-interface (GUI) front-end is called the event-flow model that represents events and event interactions. In much the same way as a control-flow model represents all possible execution paths in a program, and a data-flow model represents all possible definitions and uses of a memory location, the event-flow model represents all possible sequences of events that can be executed on the GUI. Morespecifically, a GUI is decomposed into a hierarchy of modal dialogs; this hierarchy is represented as an integration tree; each modal dialog is represented as an event-flow graph that shows all possible event execution paths in the dialog; individual events are represented using their preconditions and effects. An overview of the event-flow model with associated algorithms to semi-automatically reverse engineer the model from an executing GUI software is presented in this 2007 paper. Because the event-flow model is not tied to a specific aspect of the GUI testing process, it may be used to perform a wide variety of testing tasks by defining specialized model-based techniques called event-space exploration strategies (ESES). These ESES use the event-flow model in a number of ways to develop an end-to-end GUI testing process, namely by checking the model, test-case generation, and test oracle creation. Please see the GUI Testing page for more details.
Test case generation by using a Markov chains model
Markov chains are an efficient way to handle Model-based Testing. Test model realized with Markov chains model can be understood as a usage model: we spoke of Usage/Statistical Model Based Testing. Usage models, so Markov chains, are mainly constructed by 2 artifacts : the Finite State MachineFinite state machine
A finite-state machine or finite-state automaton , or simply a state machine, is a mathematical model used to design computer programs and digital logic circuits. It is conceived as an abstract machine that can be in one of a finite number of states...
(FSM) which represents all possible usage scenario of the system and the Operational Profiles (OP) which qualify the FSM to represent how the system is or will be used statistically. The first (FSM) helps to know what can be or has been tested and the second (OP) helps to derive operational test cases.
Usage/Statistical Model-based Testing starts from the facts that is not not possible to exhaustively test a system and that failure can appear with a very low rate. This approach offers a pragmatic way to statically derive test cases focused on: improving as prompt as possible the system under test reliability.
See also
- Domain Specific Language (DSL)
- Domain Specific ModelingDomain-Specific ModelingDomain-specific modeling is a software engineering methodology for designing and developing systems, such as computer software. It involves systematic use of a domain-specific language to represent the various facets of a system...
(DSM) - Model Driven Architecture (MDA)
- Model Driven Engineering (MDE)
- Object Oriented Analysis and DesignObject-oriented analysis and designObject-oriented analysis and design is a software engineering approach that models a system as a group of interacting objects. Each object represents some entity of interest in the system being modeled, and is characterised by its class, its state , and its behavior...
(OOAD)
Further reading
- OMG UML 2 Testing Profile; http://www.omg.org/cgi-bin/doc?formal/05-07-07.pdf
- Eckard Bringmann, Andreas Krämer; Model-based Testing of Automotive Systems In: ICST, pp. 485–493, 2008 International Conference on Software Testing, Verification, and Validation, 2008.
- Practical Model-Based Testing: A Tools Approach, Mark Utting and Bruno Legeard, ISBN 978-0-12-372501-1, Morgan-Kaufmann 2007.
- Model-Based Software Testing and Analysis with C#, Jonathan Jacky, Margus Veanes, Colin Campbell, and Wolfram Schulte, ISBN 978-0-521-68761-4, Cambridge University Press 2008.
- Model-Based Testing of Reactive Systems Advanced Lecture Series, LNCS 3472, Springer-Verlag, 2005.
- Requirements for information systems model-based testing
- Model-Based Testing Adds Value, Ewald Roodenrijs, Methods & Tools, Spring 2010.
- A Systematic Review of Model Based Testing Tool Support, Muhammad Shafique, Yvan Labiche, Carleton University, Technical Report, May 2010.
- Model-Based Testing for Embedded Systems (Computational Analysis, Synthesis, and Design of Dynamic Systems), Justyna Zander, Ina Schieferdecker, Pieter J. Mosterman, 592 pages, CRC Press, ISBN 1439818452, September 15, 2011.
- Online Community for Model-based Testing