Cloud testing
Encyclopedia
Cloud testing is a form of software testing
in which web applications use cloud computing
environments (a "cloud") to simulate real-world user traffic.
Organizations pursuing testing in general and load, performance testing and production service monitoring in particular are challenged by several problems like limited test budget, meeting deadlines. High costs per test, large number of test cases, and little or no reuse of tests and geographical distribution of users add to the challenges. Moreover ensuring high quality service delivery and avoiding outages requires testing in one's datacenter, outside the data-center, or both. Cloud Testing is the solution to all these problems. Effective unlimited storage, quick availability of the infrastructure with scalability, flexibility and availability of distributed testing environment reduce the execution time of testing of large applications and lead to cost-effective solutions.
Cloud testing is a new approach which helps us to measure accurate performance by externally load testing these applications and overcome the limitations of traditional testing.
assures this by creating peak loads using simulators.But the cost of creating such scenarios is enormous. Instead of investing capital in building on-premise testing environments, cloud testing offers affordable and scalable alternative.
of an application involves creation of heavy user traffic, and measuring its response. There is also a need to tune the performance of any application to meet certain standards.
. For this, testing performance under a particular workload is necessary. By using cloud testing, it is easy to create such environment and vary the nature of traffic on-demand. This effectively reduces cost and time by simulating thousands of geographically targeted users.
Functional testing
, latency testing, browser performance, compatibility testing are some other types of testing that can be performed in the cloud.
, HP, Load Impact
, Compuware
and Keynote systems
. Once user scenarios are developed and the test is designed, these service providers leverage cloud servers (provided by cloud platform vendors such as Amazon.com
, Google
, Rackspace
, etc.) to generate Web traffic
that originates from around the world. Once the test is complete, the cloud service providers deliver results and analytics back to corporate IT professionals through real-time dashboards for a complete analysis of how their applications and the internet will perform during peak volumes.
(SaaS) and Utility Computing
. In regard to test execution, the software offered as a service may be a transaction generator and the cloud provider's infrastructure software, or may just be the latter. Distributed Systems and Parallel Systems
mainly use this approach for testing, because of their inherent complex nature. D-Cloud is a software testing environment which uses Cloud Testing for designing Fault-tolerant systems.
Some of the tools for cloud testing include :
for software testing
purposes has been an inhibitor to overall web reliability. The low cost and accessibility of the cloud's extremely large computing resources provides the ability to replicate real world usage of these systems by geographically distributed users, executing wide varieties of user scenarios, at scales previously unattainable in traditional testing environments. Minimal start-up time along with quality assurance can be achieved by cloud testing.
Following are some of the key benefits:
Legacy systems & services need to be modified in order to be tested on cloud.Usage of robust interfaces with these legacy systems may solve this problem. Also like any other cloud services, cloud testing is vulnerable to security issues.
The test results may not be accurate due to varying performance of service providers’ network and internet.
Software testing
Software testing is an investigation conducted to provide stakeholders with information about the quality of the product or service under test. Software testing can also provide an objective, independent view of the software to allow the business to appreciate and understand the risks of software...
in which web applications use cloud computing
Cloud computing
Cloud computing is the delivery of computing as a service rather than a product, whereby shared resources, software, and information are provided to computers and other devices as a utility over a network ....
environments (a "cloud") to simulate real-world user traffic.
Overview
Cloud Testing uses cloud infrastructure for software testing.Organizations pursuing testing in general and load, performance testing and production service monitoring in particular are challenged by several problems like limited test budget, meeting deadlines. High costs per test, large number of test cases, and little or no reuse of tests and geographical distribution of users add to the challenges. Moreover ensuring high quality service delivery and avoiding outages requires testing in one's datacenter, outside the data-center, or both. Cloud Testing is the solution to all these problems. Effective unlimited storage, quick availability of the infrastructure with scalability, flexibility and availability of distributed testing environment reduce the execution time of testing of large applications and lead to cost-effective solutions.
Need for Cloud testing
Today majority of Applications run on internet and depend on good performance of the network.Causes of performance degradation
The following factors cause degradation in the performance.- Unpredictable rate of increase in number of users
- Fast paced addition of new features to existing applications
- Changes in the third party allied software
- Variation in deployment environment depending on client requirements
Limitations of traditional testing techniques
Traditional approach to test these applications use internal testing techniques that have the following limitations.- Testing at web scaleWeb testingWeb testing is the name given to software testing that focuses on web applications. Complete testing of a web-based system before going live can help address issues before the system is revealed to the public...
- Testing firewalls and load balancers
- Simulation of user activity from different geographic locations
- Testing external components
- High Cost of hardware, software and its maintenance
Cloud testing is a new approach which helps us to measure accurate performance by externally load testing these applications and overcome the limitations of traditional testing.
Stress test
Stress Test is used to determine ability of application to maintain a certain level of effectiveness beyond breaking point. It is essential for any application to work even under excessive stress and maintain stability. Stress testingStress testing
Stress testing is a form of testing that is used to determine the stability of a given system or entity. It involves testing beyond normal operational capacity, often to a breaking point, in order to observe the results...
assures this by creating peak loads using simulators.But the cost of creating such scenarios is enormous. Instead of investing capital in building on-premise testing environments, cloud testing offers affordable and scalable alternative.
Load Test
Load testingLoad testing
Load testing is the process of putting demand on a system or device and measuring its response. Load testing is performed to determine a system’s behavior under both normal and anticipated peak load conditions. It helps to identify the maximum operating capacity of an application as well as any...
of an application involves creation of heavy user traffic, and measuring its response. There is also a need to tune the performance of any application to meet certain standards.
Performance Test
Finding out thresholds, bottlenecks & limitations is a part of performance testingPerformance testing
In software engineering, performance testing is in general testing performed to determine how a system performs in terms of responsiveness and stability under a particular workload...
. For this, testing performance under a particular workload is necessary. By using cloud testing, it is easy to create such environment and vary the nature of traffic on-demand. This effectively reduces cost and time by simulating thousands of geographically targeted users.
Functional testing
Functional testing
Functional testing is a type of black box testing that bases its test cases on the specifications of the software component under test. Functions are tested by feeding them input and examining the output, and internal program structure is rarely considered .Functional testing differs from system...
, latency testing, browser performance, compatibility testing are some other types of testing that can be performed in the cloud.
Steps
Companies simulate real world Web users by using cloud testing services that are provided by cloud service vendors such as SOASTASOASTA
SOASTA, Inc. is a privately-heldAmerican technology company that provides services to test websites and web applications.It is the leading provider of cloud-based testing services,and created the industry's first browser-based website testing product....
, HP, Load Impact
Load Impact
Load Impact is a cloud-based service for load testing and website optimization. It generates traffic load by simulating users. Load Impact is a so-called freemium service in that it offers a free option as well as a premium option, where the latter gives access to more features...
, Compuware
Compuware
Compuware Corporation is a software company with products aimed at the information technology departments of large businesses. The company's services also include testing, development, professional services automation, project and portfolio management, cloud-based collaboration and performance...
and Keynote systems
Keynote systems
Keynote Systems is a US-based Internet company that specializes in developing and marketing SaaS technology solutions to measure, test, improve, from the end user perspective, the performance of websites, online content, applications, and services across browsers, networks, and mobile devices...
. Once user scenarios are developed and the test is designed, these service providers leverage cloud servers (provided by cloud platform vendors such as Amazon.com
Amazon.com
Amazon.com, Inc. is a multinational electronic commerce company headquartered in Seattle, Washington, United States. It is the world's largest online retailer. Amazon has separate websites for the following countries: United States, Canada, United Kingdom, Germany, France, Italy, Spain, Japan, and...
Google
Google Inc. is an American multinational public corporation invested in Internet search, cloud computing, and advertising technologies. Google hosts and develops a number of Internet-based services and products, and generates profit primarily from advertising through its AdWords program...
, Rackspace
Rackspace
Rackspace US, Inc. is an IT hosting company based in San Antonio, Texas. The company also has offices in Australia, the United Kingdom, The Netherlands and Hong Kong, and data centers operating in Texas, Illinois, Virginia, the United Kingdom, and Hong Kong in late 2008...
, etc.) to generate Web traffic
Web traffic
Web traffic is the amount of data sent and received by visitors to a web site. It is a large portion of Internet traffic. This is determined by the number of visitors and the number of pages they visit...
that originates from around the world. Once the test is complete, the cloud service providers deliver results and analytics back to corporate IT professionals through real-time dashboards for a complete analysis of how their applications and the internet will perform during peak volumes.
Keys to successful testing
- Understanding a platform provider's elasticity model/dynamic configuration method
- Staying abreast of the provider's evolving monitoring services and Service Level AgreementService Level AgreementA service-level agreement is a part of a service contract where the level of service is formally defined. In practice, the term SLA is sometimes used to refer to the contracted delivery time or performance...
s (SLAs) - Potentially engaging the service provider as an on-going operations partner if producing commercial off-the-shelfCommercial off-the-shelfIn the United States, Commercially available Off-The-Shelf is a Federal Acquisition Regulation term defining a nondevelopmental item of supply that is both commercial and sold in substantial quantities in the commercial marketplace, and that can be procured or utilized under government contract...
(COTS) software - Being willing to be used as a case study by the cloud service provider. The latter may lead to cost reductions.
Applications
Cloud testing is often seen as only performance or load tests, however, as discussed earlier it covers many other types of testing. Cloud computing itself is often referred to as the marriage of Software as a ServiceSoftware as a Service
Software as a service , sometimes referred to as "on-demand software," is a software delivery model in which software and its associated data are hosted centrally and are typically accessed by users using a thin client, normally using a web browser over the Internet.SaaS has become a common...
(SaaS) and Utility Computing
Utility computing
Utility computing is the packaging of computing resources, such as computation, storage and services, as a metered service similar to a traditional public utility...
. In regard to test execution, the software offered as a service may be a transaction generator and the cloud provider's infrastructure software, or may just be the latter. Distributed Systems and Parallel Systems
Parallel computing
Parallel computing is a form of computation in which many calculations are carried out simultaneously, operating on the principle that large problems can often be divided into smaller ones, which are then solved concurrently . There are several different forms of parallel computing: bit-level,...
mainly use this approach for testing, because of their inherent complex nature. D-Cloud is a software testing environment which uses Cloud Testing for designing Fault-tolerant systems.
Tools
Leading cloud computing service providers include, among others, Amazon, 3-terra, Skytap, HP and SOASTA.Some of the tools for cloud testing include :
- SOASTASOASTASOASTA, Inc. is a privately-heldAmerican technology company that provides services to test websites and web applications.It is the leading provider of cloud-based testing services,and created the industry's first browser-based website testing product....
- Cloud Assault
- Parasoft® SOAtest™SoatestParasoft SOAtest is "a comprehensive testing and analysis tool suite for tailored to the unique testing and validation needs of Service Oriented Architectures." It helps testers test the message layer functionality of their services by automating their testing and supports numerous transport...
- HP LoadRunner
- Keynote
- LoadStorm™
- Cloud Testing
Benefits
The ability and cost to simulate web trafficWeb traffic
Web traffic is the amount of data sent and received by visitors to a web site. It is a large portion of Internet traffic. This is determined by the number of visitors and the number of pages they visit...
for software testing
Software testing
Software testing is an investigation conducted to provide stakeholders with information about the quality of the product or service under test. Software testing can also provide an objective, independent view of the software to allow the business to appreciate and understand the risks of software...
purposes has been an inhibitor to overall web reliability. The low cost and accessibility of the cloud's extremely large computing resources provides the ability to replicate real world usage of these systems by geographically distributed users, executing wide varieties of user scenarios, at scales previously unattainable in traditional testing environments. Minimal start-up time along with quality assurance can be achieved by cloud testing.
Following are some of the key benefits:
- Reduction in capital expenditureCapital expenditureCapital expenditures are expenditures creating future benefits. A capital expenditure is incurred when a business spends money either to buy fixed assets or to add to the value of an existing fixed asset with a useful life extending beyond the taxable year...
- Highly scalable
Issues
The initial setup cost for migrating testing to cloud is very high as it involves modifying some of the test cases to suit cloud environment. Thus, making the decision of migration crucial. Cloud testing is not necessarily the best solution to all testing problems.Legacy systems & services need to be modified in order to be tested on cloud.Usage of robust interfaces with these legacy systems may solve this problem. Also like any other cloud services, cloud testing is vulnerable to security issues.
The test results may not be accurate due to varying performance of service providers’ network and internet.