Acceptance test
Encyclopedia
In engineering
and its various subdisciplines, acceptance testing is a test conducted to determine if the requirements of a specification or contract
are met. It may involve chemical test
s, physical test
s, or performance test
s.
In systems engineering
it may involve black-box testing performed on a system
(for example: a piece of software, lots of manufactured mechanical parts, or batches of chemical products) prior to its delivery. It is also known as functional testing
, black-box testing, QA testing, application testing, confidence testing, final testing, validation testing, or factory acceptance testing.
Software developers often distinguish acceptance testing by the system provider from acceptance testing by the customer
(the user or client) prior to accepting transfer of ownership. In the case of software, acceptance testing performed by the customer is known as user acceptance testing (UAT), end-user testing, site (acceptance) testing, or field (acceptance) testing.
A smoke test
is used as an acceptance test prior to introducing a build to the main testing process.
, outcome. There is generally no degree of success or failure. The test environment is usually designed to be identical, or as close as possible, to the anticipated user's environment, including extremes of such. These test cases must each be accompanied by test case input data or a formal description of the operational activities (or both) to be performed—intended to thoroughly exercise the specific case—and a formal description of the expected results.
Acceptance Tests/Criteria (in Agile Software Development
) are usually created by business customers and expressed in a business domain language. These are high-level tests to test the completeness of a user story
or stories 'played' during any sprint/iteration. These tests are created ideally through collaboration between business customers, business analysts, testers and developers, however the business customers (product owners) are the primary owners of these tests. As the user stories pass their acceptance criteria, the business owners can be sure of the fact that the developers are progressing in the right direction about how the application was envisaged to work and so it's essential that these tests include both business logic tests as well as UI validation elements (if need be).
Acceptance test cards are ideally created during sprint planning or iteration planning meeting, before development begins so that the developers have a clear idea of what to develop. Sometimes (due to bad planning!) acceptance tests may span multiple stories (that are not implemented in the same sprint) and there are different ways to test them out during actual sprints. One popular technique is to mock external interfaces or data to mimic other stories which might not be played out during an iteration (as those stories may have been relatively lower business priority). A user story is not considered complete until the acceptance tests have passed.
The objective is to provide confidence that the delivered system meets the business requirements of both sponsors and users. The acceptance phase may also act as the final quality gateway, where any quality defects not previously detected may be uncovered.
A principal purpose of acceptance testing is that, once completed successfully, and provided certain additional (contractually agreed) acceptance criteria are met, the sponsors will then sign off on the system as satisfying the contract
(previously agreed between sponsor and manufacturer), and deliver final payment.
(SME), preferably the owner or client of the object under test, provides such confirmation after trial or review. In software development
, UAT is one of the final stages of a project and often occurs before a client or customer accepts the new system.
Users of the system perform these tests, which developers derive from the client's contract or the user requirements specification
.
Test-designers draw up formal tests and devise a range of severity levels. Ideally the designer of the user acceptance tests should not be the creator of the formal integration and system test case
s for the same system. The UAT acts as a final verification of the required business function and proper functioning of the system, emulating real-world usage conditions on behalf of the paying client or a specific large customer. If the software works as intended and without issues during normal use, one can reasonably extrapolate the same level of stability in production.
User tests, which are usually performed by clients or end-users, do not normally focus on identifying simple problems such as spelling errors and cosmetic problems, nor showstopper
defects, such as software crashes
; testers and developers previously identify and fix these issues during earlier unit testing, integration testing
, and system testing
phases.
The results of these tests give confidence to the clients as to how the system will perform in production. There may also be legal or contractual requirements for acceptance of the system.
is a pre-requisite. This methodology - if carried out correctly - results in a quick turnaround against plan, a decreased number of test scenarios which are more complex and wider in breadth than traditional UAT and ultimately the equivalent confidence-level attained via a shorter delivery-window, allowing products/changes to come to market quicker.
The Q-UAT approach depends on a "gated" three-dimensional model. The key concepts are:
The four "gates" which conjoin and support the 3-dimensional model act as quality safeguards and include contemporary testing concepts such as:
The Quantified Approach was shaped by the former "guerilla" method of acceptance testing which was itself a response to testing phases which proved too costly to be sustainable for many small/medium-scale projects.
methodologies, particularly Extreme Programming
, referring to the functional testing of a user story
by the software development team during the implementation phase.
The customer specifies scenarios to test when a user story has been correctly implemented. A story can have one or many acceptance tests, whatever it takes to ensure the functionality works. Acceptance tests are black box system tests. Each acceptance test represents some expected result from the system. Customers are responsible for verifying the correctness of the acceptance tests and reviewing test scores to decide which failed tests are of highest priority. Acceptance tests are also used as regression tests prior to a production release. A user story is not considered complete until it has passed its acceptance tests. This means that new acceptance tests must be created for each iteration or the development team will report zero progress.
User acceptance testing:
Operational Acceptance Testing (OAT):Also known as operational readiness testing, this refers to the checking done to a system to ensure that processes and procedures are in place to allow the system to be used and maintained. This may include checks done to back-up facilities, procedures for disaster recovery, training for end users, maintenance procedures, and security procedures.
Contract and regulation acceptance testing
Alpha and beta testing
Engineering
Engineering is the discipline, art, skill and profession of acquiring and applying scientific, mathematical, economic, social, and practical knowledge, in order to design and build structures, machines, devices, systems, materials and processes that safely realize improvements to the lives of...
and its various subdisciplines, acceptance testing is a test conducted to determine if the requirements of a specification or contract
Contract
A contract is an agreement entered into by two parties or more with the intention of creating a legal obligation, which may have elements in writing. Contracts can be made orally. The remedy for breach of contract can be "damages" or compensation of money. In equity, the remedy can be specific...
are met. It may involve chemical test
Chemical test
In chemistry, a chemical test is a qualitative or quantitative procedure designed to prove the existence of, or to quantify, a chemical compound or chemical group with the aid of a specific reagent...
s, physical test
Physical test
A physical test is a qualitative or quantitative procedure that consists of determination of one or more characteristics of a given product, process or service according to a specified procedure...
s, or performance test
Performance test
The performance test or "PT" is a section of the bar exam that is intended to mimic a real-life legal task that future lawyers may face. Of the three parts of most states' bar exams -- MBE, essay, and PT -- the PT is probably the most reflective of how well a candidate will perform outside of an...
s.
In systems engineering
Systems engineering
Systems engineering is an interdisciplinary field of engineering that focuses on how complex engineering projects should be designed and managed over the life cycle of the project. Issues such as logistics, the coordination of different teams, and automatic control of machinery become more...
it may involve black-box testing performed on a system
System
System is a set of interacting or interdependent components forming an integrated whole....
(for example: a piece of software, lots of manufactured mechanical parts, or batches of chemical products) prior to its delivery. It is also known as functional testing
Functional testing
Functional testing is a type of black box testing that bases its test cases on the specifications of the software component under test. Functions are tested by feeding them input and examining the output, and internal program structure is rarely considered .Functional testing differs from system...
, black-box testing, QA testing, application testing, confidence testing, final testing, validation testing, or factory acceptance testing.
Software developers often distinguish acceptance testing by the system provider from acceptance testing by the customer
Customer
A customer is usually used to refer to a current or potential buyer or user of the products of an individual or organization, called the supplier, seller, or vendor. This is typically through purchasing or renting goods or services...
(the user or client) prior to accepting transfer of ownership. In the case of software, acceptance testing performed by the customer is known as user acceptance testing (UAT), end-user testing, site (acceptance) testing, or field (acceptance) testing.
A smoke test
Smoke test
Smoke testing refers to physical tests made to closed systems of pipes to test for leaks. By metaphorical extension, the term is also used for the first test made after assembly or repairs to a system, to provide some assurance that the system under test will not catastrophically fail...
is used as an acceptance test prior to introducing a build to the main testing process.
Overview
Testing generally involves running a suite of tests on the completed system. Each individual test, known as a case, exercises a particular operating condition of the user's environment or feature of the system, and will result in a pass or fail, or booleanBoolean logic
Boolean algebra is a logical calculus of truth values, developed by George Boole in the 1840s. It resembles the algebra of real numbers, but with the numeric operations of multiplication xy, addition x + y, and negation −x replaced by the respective logical operations of...
, outcome. There is generally no degree of success or failure. The test environment is usually designed to be identical, or as close as possible, to the anticipated user's environment, including extremes of such. These test cases must each be accompanied by test case input data or a formal description of the operational activities (or both) to be performed—intended to thoroughly exercise the specific case—and a formal description of the expected results.
Acceptance Tests/Criteria (in Agile Software Development
Agile software development
Agile software development is a group of software development methodologies based on iterative and incremental development, where requirements and solutions evolve through collaboration between self-organizing, cross-functional teams...
) are usually created by business customers and expressed in a business domain language. These are high-level tests to test the completeness of a user story
User story
In computer programming a user story is one or more sentences in the everyday or business language of the end user that captures what the user wants to achieve. User stories are used with Agile software development methodologies for the basis of what features that can be implemented...
or stories 'played' during any sprint/iteration. These tests are created ideally through collaboration between business customers, business analysts, testers and developers, however the business customers (product owners) are the primary owners of these tests. As the user stories pass their acceptance criteria, the business owners can be sure of the fact that the developers are progressing in the right direction about how the application was envisaged to work and so it's essential that these tests include both business logic tests as well as UI validation elements (if need be).
Acceptance test cards are ideally created during sprint planning or iteration planning meeting, before development begins so that the developers have a clear idea of what to develop. Sometimes (due to bad planning!) acceptance tests may span multiple stories (that are not implemented in the same sprint) and there are different ways to test them out during actual sprints. One popular technique is to mock external interfaces or data to mimic other stories which might not be played out during an iteration (as those stories may have been relatively lower business priority). A user story is not considered complete until the acceptance tests have passed.
Process
The acceptance test suite is run against the supplied input data or using an acceptance test script to direct the testers. Then the results obtained are compared with the expected results. If there is a correct match for every case, the test suite is said to pass. If not, the system may either be rejected or accepted on conditions previously agreed between the sponsor and the manufacturer.The objective is to provide confidence that the delivered system meets the business requirements of both sponsors and users. The acceptance phase may also act as the final quality gateway, where any quality defects not previously detected may be uncovered.
A principal purpose of acceptance testing is that, once completed successfully, and provided certain additional (contractually agreed) acceptance criteria are met, the sponsors will then sign off on the system as satisfying the contract
Contract
A contract is an agreement entered into by two parties or more with the intention of creating a legal obligation, which may have elements in writing. Contracts can be made orally. The remedy for breach of contract can be "damages" or compensation of money. In equity, the remedy can be specific...
(previously agreed between sponsor and manufacturer), and deliver final payment.
User acceptance testing
User Acceptance Testing (UAT) is a process to obtain confirmation that a system meets mutually agreed-upon requirements. A Subject Matter ExpertSubject Matter Expert
A subject matter expert or domain expert is a person who is an expert in a particular area or topic. When spoken, sometimes the acronym "SME" is spelled out and other times voiced as a word ....
(SME), preferably the owner or client of the object under test, provides such confirmation after trial or review. In software development
Software development process
A software development process, also known as a software development life cycle , is a structure imposed on the development of a software product. Similar terms include software life cycle and software process. It is often considered a subset of systems development life cycle...
, UAT is one of the final stages of a project and often occurs before a client or customer accepts the new system.
Users of the system perform these tests, which developers derive from the client's contract or the user requirements specification
Requirements analysis
Requirements analysis in systems engineering and software engineering, encompasses those tasks that go into determining the needs or conditions to meet for a new or altered product, taking account of the possibly conflicting requirements of the various stakeholders, such as beneficiaries or users...
.
Test-designers draw up formal tests and devise a range of severity levels. Ideally the designer of the user acceptance tests should not be the creator of the formal integration and system test case
Test case
A test case in software engineering is a set of conditions or variables under which a tester will determine whether an application or software system is working correctly or not. The mechanism for determining whether a software program or system has passed or failed such a test is known as a test...
s for the same system. The UAT acts as a final verification of the required business function and proper functioning of the system, emulating real-world usage conditions on behalf of the paying client or a specific large customer. If the software works as intended and without issues during normal use, one can reasonably extrapolate the same level of stability in production.
User tests, which are usually performed by clients or end-users, do not normally focus on identifying simple problems such as spelling errors and cosmetic problems, nor showstopper
Showstopper
Showstopper may refer to:* "Show Stopper", the debut single by the R&B group and Making the Band 3 winners Danity Kane...
defects, such as software crashes
Crash (computing)
A crash in computing is a condition where a computer or a program, either an application or part of the operating system, ceases to function properly, often exiting after encountering errors. Often the offending program may appear to freeze or hang until a crash reporting service documents...
; testers and developers previously identify and fix these issues during earlier unit testing, integration testing
Integration testing
Integration testing is the phase in software testing in which individual software modules are combined and tested as a group. It occurs after unit testing and before validation testing...
, and system testing
System testing
System testing of software or hardware is testing conducted on a complete, integrated system to evaluate the system's compliance with its specified requirements. System testing falls within the scope of black box testing, and as such, should require no knowledge of the inner design of the code or...
phases.
The results of these tests give confidence to the clients as to how the system will perform in production. There may also be legal or contractual requirements for acceptance of the system.
Q-UAT - Quantified User Acceptance Testing
Quantified User Acceptance Testing (Q-UAT or, more simply, the "Quantified Approach") is a revised Business Acceptance Testing process which aims to provide a smarter and faster alternative to the traditional UAT phase. Depth-testing is carried out against business requirements only at specific planned points in the application or service under test. A reliance on better quality code-delivery from the development/build phase is assumed and a complete understanding of the appropriate business processBusiness process
A business process or business method is a collection of related, structured activities or tasks that produce a specific service or product for a particular customer or customers...
is a pre-requisite. This methodology - if carried out correctly - results in a quick turnaround against plan, a decreased number of test scenarios which are more complex and wider in breadth than traditional UAT and ultimately the equivalent confidence-level attained via a shorter delivery-window, allowing products/changes to come to market quicker.
The Q-UAT approach depends on a "gated" three-dimensional model. The key concepts are:
- Linear Testing (LT, the 1st dimension)
- Recursive Testing (RT, the 2nd dimension)
- Adaptive Testing (AT, the 3rd dimension).
The four "gates" which conjoin and support the 3-dimensional model act as quality safeguards and include contemporary testing concepts such as:
- Internal Consistency Checks (ICS)
- Major Systems/Services Checks (MSC)
- Realtime/Reactive Regression (RTR).
The Quantified Approach was shaped by the former "guerilla" method of acceptance testing which was itself a response to testing phases which proved too costly to be sustainable for many small/medium-scale projects.
Acceptance testing in Extreme Programming
Acceptance testing is a term used in agile software developmentAgile software development
Agile software development is a group of software development methodologies based on iterative and incremental development, where requirements and solutions evolve through collaboration between self-organizing, cross-functional teams...
methodologies, particularly Extreme Programming
Extreme Programming
Extreme programming is a software development methodology which is intended to improve software quality and responsiveness to changing customer requirements...
, referring to the functional testing of a user story
User story
In computer programming a user story is one or more sentences in the everyday or business language of the end user that captures what the user wants to achieve. User stories are used with Agile software development methodologies for the basis of what features that can be implemented...
by the software development team during the implementation phase.
The customer specifies scenarios to test when a user story has been correctly implemented. A story can have one or many acceptance tests, whatever it takes to ensure the functionality works. Acceptance tests are black box system tests. Each acceptance test represents some expected result from the system. Customers are responsible for verifying the correctness of the acceptance tests and reviewing test scores to decide which failed tests are of highest priority. Acceptance tests are also used as regression tests prior to a production release. A user story is not considered complete until it has passed its acceptance tests. This means that new acceptance tests must be created for each iteration or the development team will report zero progress.
Types of acceptance testing
Typical types of acceptance testing include the followingUser acceptance testing:
- This may include factory acceptance testing, i.e. the testing done by factory users before the factory is moved to its own site, after which site acceptance testing may be performed by the users at the site.
Operational Acceptance Testing (OAT):Also known as operational readiness testing, this refers to the checking done to a system to ensure that processes and procedures are in place to allow the system to be used and maintained. This may include checks done to back-up facilities, procedures for disaster recovery, training for end users, maintenance procedures, and security procedures.
Contract and regulation acceptance testing
- In contract acceptance testing, a system is tested against acceptance criteria as documented in a contract, before the system is accepted. In regulation acceptance testing, a system is tested to ensure it meets governmental, legal and safety standards.
Alpha and beta testing
- Alpha testing takes place at developers' sites, and involves testing of the operational system by internal staff, before it is released to external customers. Beta testing takes place at customers' sites, and involves testing by a group of customers who use the system at their own locations and provide feedback, before the system is released to other customers. The latter is often called “field testing”.
List of development to production (testing) environments
- Development Environment
- Development Testing Environment
- Testing Environment
- Development Integration Testing
- Development System Testing
- System Integration Testing
- User Acceptance Testing
- Production Environment
List of acceptance-testing frameworks
- FitNesseFitNesseFitNesse is a web server, a wiki, and an automated testing tool for software. It is based on Ward Cunningham's Framework for Integrated Test. FitNesse is designed to support acceptance testing rather than unit testing in that it facilitates detailed readable description of system function.FitNesse...
, a fork of Fit - Framework for Integrated Test (Fit)
- iMacrosIMacrosiMacros is an extension for the Mozilla Firefox, Google Chrome, and Internet Explorer web browsers which adds record and replay functionality similar to that found in web testing and form filler software. The macros can be combined and controlled via JavaScript. Demo macros and JavaScript code...
- ItsNatItsNatItsNat Natural AJAX, is an open source Java component-based Ajax framework.- Overview :ItsNat is server centric using an approach called the browser is the server...
Java Ajax web framework with built-in, server based, functional web testing capabilities. - RanorexRanorexRanorex is a Windows GUI test automation framework for testing many different application types including Web 2.0 applications, Win32, MFC, WPF, Flash/Flex, .NET and Java . Ranorex doesn't have a scripting language of its own. The user Ranorex is a Windows GUI test automation framework for testing...
- Selenium (software)Selenium (software)Selenium is a portable software testing framework for web applications. Selenium provides a record/playback tool for authoring tests without learning a test scripting language . It also provides a test domain-specific language to write tests in a number of popular programming languages, including...
- Test Automation FXTest Automation FXTest Automation FX is a Windows GUI test automation framework for Visual Studio 2008 and Visual Studio 2010.Test Automation FX requires no special knowledge when it comes to custom-made scripting language, since it stores all test logic in .NET code...
- WatirWatirWeb Application Testing in Ruby is a toolkit used to automate browser-based tests during web application development. This automated test tool uses the Ruby scripting language to drive Internet Explorer, Mozilla Firefox, Google Chrome, Opera and Safari, and is available as a RubyGems gem...
- Fabasoft app.testFabasoft app.testFabasoft app.testFabasoft app.test is a test tool for the automated quality assurance of web applications in Microsoft Windows, Ubuntu Linux and Mac OS X. In a matter of clicks, easy to understand tests can be carried out and recorded on your computer. The running of tests can also be fully...
for automated acceptance tests
See also
- Acceptance samplingAcceptance samplingAcceptance sampling uses statistical sampling to determine whether to accept or reject a production lot of material. It has been a common quality control technique used in industry and particularly the military for contracts and procurement. It is usually done as products leave the factory, or in...
- Black box testingBlack box testingBlack-box testing is a method of software testing that tests the functionality of an application as opposed to its internal structures or workings . Specific knowledge of the application's code/internal structure and programming knowledge in general is not required...
- Development stageDevelopment stageA software release life cycle refers to the phases of development and maturity for a piece of computer software—ranging from its initial development, to its eventual release, and updated versions of the released version to help improve software or fix bugs still present in the software.- Pre-alpha...
- Dynamic testingDynamic testingDynamic testing is a term used in software engineering to describe the testing of the dynamic behavior of code. That is, dynamic analysis refers to the examination of the physical response from the system to variables that are not constant and change with time...
- Grey box testing
- Software testingSoftware testingSoftware testing is an investigation conducted to provide stakeholders with information about the quality of the product or service under test. Software testing can also provide an objective, independent view of the software to allow the business to appreciate and understand the risks of software...
- System testingSystem testingSystem testing of software or hardware is testing conducted on a complete, integrated system to evaluate the system's compliance with its specified requirements. System testing falls within the scope of black box testing, and as such, should require no knowledge of the inner design of the code or...
- Test-driven developmentTest-driven developmentTest-driven development is a software development process that relies on the repetition of a very short development cycle: first the developer writes a failing automated test case that defines a desired improvement or new function, then produces code to pass that test and finally refactors the new...
- Unit testing
- White box testingWhite box testingWhite-box testing is a method of testing software that tests internal structures or workings of an application, as opposed to its functionality...