Multivariate testing
Encyclopedia
In statistics
, multivariate testing or multi-variable testing is a technique for testing hypotheses on complex multi-variable systems, especially used in testing market perceptions.
, multivariate testing is a process by which more than one component of a website may be tested in a live environment. It can be thought of in simple terms as numerous A/B tests
performed on one page at the same time. A/B tests are usually performed to determine the better of two content variations; multivariate testing can theoretically test the effectiveness of limitless combinations. The only limits on the number of combinations and the number of variables in a multivariate test are the amount of time it will take to get a statistically valid sample of visitors and computational power.
Multivariate testing is usually employed in order to ascertain which content or creative variation produces the best improvement in the defined goals of a website, whether that be user registrations or successful completion of a checkout process (that is, conversion rate
). Dramatic increases can be seen through testing different copy text, form layouts and even landing page images and background colours. However, not all elements produce the same increase in conversions, and by looking at the results from different tests, it is possible to identify those elements that consistently tend to produce the greatest increase in conversions.
Testing can be carried out on a dynamically generated website by setting up the server to display the different variations of content in equal proportions to incoming visitors. Statistics on how each visitor went on to behave after seeing the content under test must then be gathered and presented. Outsourced services can also be used to provide multivariate testing on websites with minor changes to page coding. These services insert their content to predefined areas of a site and monitor user behavior.
In a nutshell, multivariate testing can be seen as allowing website visitors to vote with their clicks for which content they prefer and will stand the most chance of their proceeding to a defined goal. The testing is transparent to the visitor with all commercial solutions capable of ensuring that each visitor is shown the same content on every visit.
Some websites benefit from constant 24/7 continuous optimization as visitor response to creatives and layouts differ by time of day/week or even season.
Multivariate testing is currently an area of high growth in internet marketing as it helps website owners to ensure that they are getting the most from the visitors arriving at their site. Areas such as search engine optimization
and pay per click
advertising bring visitors to a site and have been extensively used by many organisations but multivariate testing allows internet marketeers to ensure that visitors are being shown the right offers, content and layout to convert them to sale, registration or the desired action once they arrive at the website.
There are two principal approaches used to achieve multivariate testing on websites. One being Page Tagging
; a process where the website creator inserts Javascript
into the site to inject content variants and monitor visitor response. Page tagging typically tracks what a visitor viewed on the website and for how long that visitor remained on the site together with any click or conversion related actions performed. Page tagging usually needs to be done by a technical team and typically cannot be accomplished by a web marketer.. Later refinements on this method allow for a single common tag to be deployed across all pages, reducing deployment time and removing the need for re-deployment between tests.
Companies known to employ a tag based method of multivariate testing are: Monetate, TraceAd Analytics, Avenseo, Conversion Works, Adobe
, Business Intelligence Group GmbH (B.I.G.), Amadesa, DIVOLUTION, Maxymiser, Webtrends
Optimize, Conversion Voodoo, Google Website Optimizer, Vertster and Autonomy Corporation
The second principal approach used does not require page tagging. By establishing a DNS-proxy or hosting within a website's own datacenter, it is possible to intercept and process all web traffic to and from the site undergoing testing, insert variants and monitor visitor response. In this case, all logic sits server rather than browser-side and after initial DNS changes are made, no further technical involvement is required from the website point of view. SiteSpect is known to employ this method of implementation.
Multivariate testing can also be applied to email body content and mobile web pages.
In addition to testing the efficacy of various creative/content executions on a website, the principles of multivariate testing can and often are used to test various offer combinations. Examples of this are testing various price points, purchase incentives, premiums, trial periods or other similar purchase incentives both individually and in combination with each other. The value of this is that marketers (both traditional and online) can use multivariate testing principles online to quickly ascertain and predict the effectiveness of offers without going through the more traditional multivariate testing methods which take significantly more time and money (focus groups, telephone surveys, etc.).
. Several methods in use for multivariate testing include:
Statistics
Statistics is the study of the collection, organization, analysis, and interpretation of data. It deals with all aspects of this, including the planning of data collection in terms of the design of surveys and experiments....
, multivariate testing or multi-variable testing is a technique for testing hypotheses on complex multi-variable systems, especially used in testing market perceptions.
In internet marketing
In internet marketingInternet marketing
Internet marketing, also known as digital marketing, web marketing, online marketing, search marketing or e-marketing, is referred to as the marketing of products or services over the Internet...
, multivariate testing is a process by which more than one component of a website may be tested in a live environment. It can be thought of in simple terms as numerous A/B tests
A/B testing
A/B testing, split testing or bucket testing is a method of marketing testing by which a baseline control sample is compared to a variety of single-variable test samples in order to improve response rates...
performed on one page at the same time. A/B tests are usually performed to determine the better of two content variations; multivariate testing can theoretically test the effectiveness of limitless combinations. The only limits on the number of combinations and the number of variables in a multivariate test are the amount of time it will take to get a statistically valid sample of visitors and computational power.
Multivariate testing is usually employed in order to ascertain which content or creative variation produces the best improvement in the defined goals of a website, whether that be user registrations or successful completion of a checkout process (that is, conversion rate
Conversion rate
In internet marketing, conversion rate is the ratio of visitors who convert casual content views or website visits into desired actions based on subtle or direct requests from marketers, advertisers, and content creators...
). Dramatic increases can be seen through testing different copy text, form layouts and even landing page images and background colours. However, not all elements produce the same increase in conversions, and by looking at the results from different tests, it is possible to identify those elements that consistently tend to produce the greatest increase in conversions.
Testing can be carried out on a dynamically generated website by setting up the server to display the different variations of content in equal proportions to incoming visitors. Statistics on how each visitor went on to behave after seeing the content under test must then be gathered and presented. Outsourced services can also be used to provide multivariate testing on websites with minor changes to page coding. These services insert their content to predefined areas of a site and monitor user behavior.
In a nutshell, multivariate testing can be seen as allowing website visitors to vote with their clicks for which content they prefer and will stand the most chance of their proceeding to a defined goal. The testing is transparent to the visitor with all commercial solutions capable of ensuring that each visitor is shown the same content on every visit.
Some websites benefit from constant 24/7 continuous optimization as visitor response to creatives and layouts differ by time of day/week or even season.
Multivariate testing is currently an area of high growth in internet marketing as it helps website owners to ensure that they are getting the most from the visitors arriving at their site. Areas such as search engine optimization
Search engine optimization
Search engine optimization is the process of improving the visibility of a website or a web page in search engines via the "natural" or un-paid search results...
and pay per click
Pay per click
Pay per click is an Internet advertising model used to direct traffic to websites, where advertisers pay the publisher when the ad is clicked. With search engines, advertisers typically bid on keyword phrases relevant to their target market...
advertising bring visitors to a site and have been extensively used by many organisations but multivariate testing allows internet marketeers to ensure that visitors are being shown the right offers, content and layout to convert them to sale, registration or the desired action once they arrive at the website.
There are two principal approaches used to achieve multivariate testing on websites. One being Page Tagging
Tag (metadata)
In online computer systems terminology, a tag is a non-hierarchical keyword or term assigned to a piece of information . This kind of metadata helps describe an item and allows it to be found again by browsing or searching...
; a process where the website creator inserts Javascript
JavaScript
JavaScript is a prototype-based scripting language that is dynamic, weakly typed and has first-class functions. It is a multi-paradigm language, supporting object-oriented, imperative, and functional programming styles....
into the site to inject content variants and monitor visitor response. Page tagging typically tracks what a visitor viewed on the website and for how long that visitor remained on the site together with any click or conversion related actions performed. Page tagging usually needs to be done by a technical team and typically cannot be accomplished by a web marketer.. Later refinements on this method allow for a single common tag to be deployed across all pages, reducing deployment time and removing the need for re-deployment between tests.
Companies known to employ a tag based method of multivariate testing are: Monetate, TraceAd Analytics, Avenseo, Conversion Works, Adobe
Adobe Systems
Adobe Systems Incorporated is an American computer software company founded in 1982 and headquartered in San Jose, California, United States...
, Business Intelligence Group GmbH (B.I.G.), Amadesa, DIVOLUTION, Maxymiser, Webtrends
WebTrends
Webtrends is a private company headquartered in Portland, Oregon, United States. It provides web analytics and other software solutions related to marketing intelligence...
Optimize, Conversion Voodoo, Google Website Optimizer, Vertster and Autonomy Corporation
Autonomy Corporation
Autonomy is a multinational enterprise software company with joint headquarters in Cambridge, United Kingdom, and San Francisco, USA and a subsidiary of Hewlett-Packard. The company uses a combination of technologies born out of research at the University of Cambridge...
The second principal approach used does not require page tagging. By establishing a DNS-proxy or hosting within a website's own datacenter, it is possible to intercept and process all web traffic to and from the site undergoing testing, insert variants and monitor visitor response. In this case, all logic sits server rather than browser-side and after initial DNS changes are made, no further technical involvement is required from the website point of view. SiteSpect is known to employ this method of implementation.
Multivariate testing can also be applied to email body content and mobile web pages.
In addition to testing the efficacy of various creative/content executions on a website, the principles of multivariate testing can and often are used to test various offer combinations. Examples of this are testing various price points, purchase incentives, premiums, trial periods or other similar purchase incentives both individually and in combination with each other. The value of this is that marketers (both traditional and online) can use multivariate testing principles online to quickly ascertain and predict the effectiveness of offers without going through the more traditional multivariate testing methods which take significantly more time and money (focus groups, telephone surveys, etc.).
Design of Experiments
Statistical testing relies on design of experimentsDesign of experiments
In general usage, design of experiments or experimental design is the design of any information-gathering exercises where variation is present, whether under the full control of the experimenter or not. However, in statistics, these terms are usually used for controlled experiments...
. Several methods in use for multivariate testing include:
- Discrete choiceDiscrete choiceIn economics, discrete choice problems involve choices between two or more discrete alternatives, such as entering or not entering the labor market, or choosing between modes of transport. Such choices contrast with standard consumption models in which the quantity of each good consumed is assumed...
and what has mutated to become choice modeling is the complex technique that won Daniel McFaddenDaniel McFaddenDaniel Little McFadden is an econometrician who shared the 2000 Nobel Memorial Prize in Economic Sciences with James Heckman ; McFadden's share of the prize was "for his development of theory and methods for analyzing discrete choice". He was the E. Morris Cox Professor of Economics at the...
the Nobel Prize in Economics in 2000. Choice modeling models how people make tradeoffs in the context of a purchase decision. By systematically varying the attributes or content elements, one can quantify their impact on outcome, such as a purchase decision. What is most important are the interaction effects uncovered, which neither the Taguchi methodsTaguchi methodsTaguchi methods are statistical methods developed by Genichi Taguchi to improve the quality of manufactured goods, and more recently also applied to, engineering, biotechnology, marketing and advertising...
nor Optimal designOptimal designOptimal designs are a class of experimental designs that are optimal with respect to some statistical criterion.In the design of experiments for estimating statistical models, optimal designs allow parameters to be estimated without bias and with minimum-variance...
solve for. - Optimal designOptimal designOptimal designs are a class of experimental designs that are optimal with respect to some statistical criterion.In the design of experiments for estimating statistical models, optimal designs allow parameters to be estimated without bias and with minimum-variance...
involves iterations and waves of testings. Optimal design allows marketers the ability not only to test the maximum number of creative permutations in the shortest period of time but also to take into account relationships, interactions, and constraints across content elements on a website. This allows one to find the optimal solution unencumbered by limitations. - Taguchi methodsTaguchi methodsTaguchi methods are statistical methods developed by Genichi Taguchi to improve the quality of manufactured goods, and more recently also applied to, engineering, biotechnology, marketing and advertising...
: with multiple variations of content in multiple locations on a website, a large number of combinations need to be statistically tested and medium/low traffic websites can take some time to get a large enough sample of visitors to decide which content gives the best performance. For example, if 3 different images are to be tested in 3 locations, there are 27 combinations to test. Taguchi methodsTaguchi methodsTaguchi methods are statistical methods developed by Genichi Taguchi to improve the quality of manufactured goods, and more recently also applied to, engineering, biotechnology, marketing and advertising...
(namely Taguchi orthogonal arrays) can be used in the design of experimentsDesign of experimentsIn general usage, design of experiments or experimental design is the design of any information-gathering exercises where variation is present, whether under the full control of the experimenter or not. However, in statistics, these terms are usually used for controlled experiments...
in order to reduce the variations but still give statistically valid results on individual content elements. Taguchi uses fractional factorial designs.
See also
- Web usabilityWeb usabilityWeb usability is the application of usability in those domains where web browsing can be considered as a general paradigm for constructing a GUI.-General:...
- Multivariate statisticsMultivariate statisticsMultivariate statistics is a form of statistics encompassing the simultaneous observation and analysis of more than one statistical variable. The application of multivariate statistics is multivariate analysis...
- A/B testingA/B testingA/B testing, split testing or bucket testing is a method of marketing testing by which a baseline control sample is compared to a variety of single-variable test samples in order to improve response rates...
- Cultural Multivariate TestingCultural Multivariate TestingIn internet marketing, cultural multivariate testing refers to multivariate testing performed on an international website in each geographically distinct market for the purpose of website localisation, i.e...
- Product optimizationProduct optimizationProduct optimization is the practice of making changes or adjustments to a product to make it more desirable.-Description:A product has a number of attributes. For example, a soda bottle can have different packaging variations, flavors, nutritional values. It is possible to optimize a product by...