Keyword research is a practice used by search engine optimization professionals to find and research actual search terms people enter into the search engines when conducting a search...
Microsoft adCenter , is the division of the Microsoft Network responsible for MSN's advertising services. Microsoft adCenter provides pay per click advertisements. This is a service aimed at people who want to advertise a product...
, which contains a set of algorithms for providing information about keywords
Keywords
Keywords are the words that are used to reveal the internal structure of an author's reasoning. While they are used primarily for rhetoric, they are also used in a strictly grammatical sense for structural composition, reasoning, and comprehension...
A search engine is an information retrieval system designed to help find information stored on a computer system. The search results are usually presented in a list and are commonly called hits. Search engines help to minimize the time required to find information and the amount of information...
queries.
The KSP was originally conceived by ZhaoHui Tang, Dylan Huang, and Wayne Guan at Microsoft adCenter Labs
Microsoft adCenter Labs
Microsoft adCenter Labs , is an applied research group at Microsoft that supports Microsoft adCenter. Microsoft adCenter , is the division of the Microsoft Network responsible for MSN's advertising services....
in May 2006. The platform aims to provide a core set of data and technology to empower search engine marketing
Search engine marketing
Search engine marketing, , is a form of Internet marketing that seeks to promote websites by increasing their visibility in search engine result pages through the use of paid placement, contextual advertising, and paid inclusion...
and keyword research efforts. The KSP uniquely delivers a standardized set of keyword technologies through a Web services model, accessible via an application programming interface
Application programming interface
An application programming interface is a source code based specification intended to be used as an interface by software components to communicate with each other...
Microsoft Excel is a proprietary commercial spreadsheet application written and distributed by Microsoft for Microsoft Windows and Mac OS X. It features calculation, graphing tools, pivot tables, and a macro programming language called Visual Basic for Applications...
add-in.
KSP API beta access is available for researchers and developers upon request from the Keyword Services Platform feedback link.
Architecture
The following components comprise the Keyword Service Platform architecture:
Keyword API. Set of standard Web services for various keyword tasks. These services are based on the Windows Communications Foundation and can be consumed by client applications (e.g., Microsoft Excel add-in) or mashups
Mashup (digital)
A digital mashup usually is in reference to:1. Digital media content containing any or all of text, graphics, audio, video and animation drawn from pre-existing sources, to create a new derivative work...
.
Provider Plug-in Framework. System that allows researchers to incorporate new algorithms or data mining
Data mining
Data mining , a relatively young and interdisciplinary field of computer science is the process of discovering new patterns from large data sets involving methods at the intersection of artificial intelligence, machine learning, statistics and database systems...
models within the Service Container. Each provider brings a specific keyword technology to the KSP architecture—for instance, keyword association, keyword extraction, or keyword classification.
Security. Component that handles permissions that pertain to provider procedures (methods) and Stored Procedure implementation.
Server Object Model. Data model that allows developers to leverage different services.
Service Container. Set of providers that support various keyword technologies. Host to all service providers and Stored Procedures, which allows parallel execution.
Shared Services. Core components, consisting of a crawler
Crawler
Crawler may refer to:* Crawler, a person who crawls or creeps along the ground* Crawler , a British rock band* Crawler-transporter, a vehicle used to move space vehicles on Earth...
, in-memory data structures, word stemming algorithms, etc. These services are used by different providers and executed by stored procedures.
Stored Procedures. Procedures for consolidating and centralizing the logic behind applications. Selected sets of these procedures are made available to users.
The .NET Framework is a software framework that runs primarily on Microsoft Windows. It includes a large library and supports several programming languages which allows language interoperability...
programming languages to create procedures that combine the use of different providers, or implement additional business logic processing based on the output from a provider.
Keyword API
The Keyword Service Platform has defined a set of APIs for each class of keyword services. These interfaces for Web services include keyword extraction (ITermExtraction), keyword categorization (ITermCategorization), keyword suggestion (ITermSuggestion), keyword forecast (ITermForecast), keyword monetization (ITermMonetization), and several others. The APIs define the signatures of each Web service.
Keyword suggestion
Keyword suggestions are handled via the ITermSuggestion interface. To find the five most closely related keywords to "BMW", the following method call may be used: GetTermSuggestion("BMW",5). The query result is shown in the following table, and by default, sorted by confidence
Confidence interval
In statistics, a confidence interval is a particular kind of interval estimate of a population parameter and is used to indicate the reliability of an estimate. It is an observed interval , in principle different from sample to sample, that frequently includes the parameter of interest, if the...
:
OriginalTerm
Term
BMW
Auto
BMW
Car
BMW
Lexus
BMW
BMW cars
BMW
BMW Z4
To view the five suggested five terms with the corresponding confidence score, a third parameter can be used to indicate that statistics should be returned: GetTermSuggestion("BMW",5,true). The query result is shown in the following table along with columns for score and support. The results are similar to those available through the Data Mining Extensions
Data Mining Extensions
Data Mining Extensions is a query language for Data Mining Models supported by Microsoft's SQL Server Analysis Services product.Like SQL, it supports a data definition language, data manipulation language and a data query language, all three with SQL-like syntax.Whereas SQL statements operate on...
SQL is a programming language designed for managing data in relational database management systems ....
. Score represents the confidence or probability; support represents the number of cases supporting the rule in the training dataset.
OriginalTerm
Term
Score
Support
BMW
Auto
0.96
10000
BMW
Car
0.89
9000
BMW
Lexus
0.89
11000
BMW
BMW cars
0.83
12000
BMW
BMW Z4
0.78
12800
To return only those terms with a high confidence score, a filter can be used on the Score column with the following method call: GetTermSuggestion("BMW",5,true,"Score>0.8"). The query result is shown in the following table. In this case, only four rows are returned, as these are the only terms that meet the criterion of the filter.
OriginalTerm
Term
Score
Support
BMW
Auto
0.96
10000
BMW
Car
0.89
9000
BMW
Lexus
0.89
11000
BMW
BMW cars
0.83
12000
When the table of terms possibly includes thousands of keywords, batch query syntax can be used. For example, suppose that the keywords are stored in myInputTermTable, and only the two most relevant terms for each keyword should be returned: GetTermSuggestion(myInputTermTable,2). The query result is shown in the following table.
OriginalTerm
Term
BMW
Auto
BMW
Car
Honda
Lexus
Honda
Sedan
Ford
Pickup
Ford
Truck
Keyword demographics
Keyword demographics are handled via the ITermDemographics interface. To obtain the demographic distribution for the keyword "Minivan", the following method call could be used: GetTermDemographics("minivan"). The query result is shown in the following table.
Term
Male
Female
0-13
13-18
18-25
25-35
35-50
50-65
65+
Minivan
0.40
0.60
0
0
0.1
0.2
0.4
0.2
0.1
Keyword monetization
Keyword monetization values specific to paid search are handled via the ITermMonetization interface. The following method call returns the KPIs for the keyword "Online bank" based on the previous week's paid search data, in the third position of sponsored listings: GetTermKPIs("online bank",TimeInterval.LastWeek,3). The result of the query is shown below, containing the input keyword, the number of clicks in the sponsored link for "Online bank", overall impressions for the keyword, position, average click-through rate (CTR), and average cost per click (CPC).
Term
Clicks
Impressions
Position
CTR
CPC
Online bank
42
2915
3
0.014
1.325
Keyword extraction
Keyword extraction is handled via the ITermExtraction interface. The following method call extracts the eight most relevant keywords from the webpage "autos.msn.com", and provides the corresponding statistics: GetTermExtraction("autos.msn.com",8,true). The result of the query is shown below, where the Score column represents the relevance of the extracted keyword to the page content, while the Support column represents the number of occurrences of a keyword on the page.
URL
Term
Score
Support
autos.msn.com
auto reviews
0.62
3
autos.msn.com
MSN autos
0.54
2
autos.msn.com
cars
0.48
5
autos.msn.com
sport cars
0.39
2
autos.msn.com
used cars
0.38
1
autos.msn.com
compare car
0.34
1
autos.msn.com
new cars
0.32
1
autos.msn.com
luxury cars
0.30
1
Sample code
The following code fragment connects to the Keyword Services Platform server and uses the keyword term forecast Web service.
using (KeywordServer server = new KeywordServer("https://ksp.microsoft.com")) {
server.UserName = "username";
server.Password = "********";
ITermForecast provider = null;
try {
server.Open;
// Context can be set if needed. It will remain during the following calls.
provider = server.GetProviderByImplementation("Microsoft.adCenterLabs.Providers.KeywordForecastProvider");
if (provider != null) {
// Single mode API
DataTable result = provider.GetTermForecast(term, -5, 3);
DisplayResults(result);
// Batch mode API
result = provider.GetTermForecast(terms, -5, 3);
DisplayResults(result);
}
}
catch (FaultException) {
// Handle fault returned from calling the proxy method
}
catch (CommunicationException) {
// Handle lost network connection error
}
catch (TimeoutException) {
// Handle time-out error
}
finally {
if (provider != null)
server.ReleaseService(provider);
}
}
Providers
Each Keyword Services Platform provider supplies a specific type of keyword technology by implementing one class of a specific keyword interface (e.g., ITermSuggestion, ITermForecast, ITermExtraction). The API defines the signature of each Web service and the format of the returned data. The KSP provider is a server-side object encapsulating a particular implementation of a keyword technology. This provider exposes its functionality through service contracts in the Windows Communication Foundation
Windows Communication Foundation
The Windows Communication Foundation , previously known as "Indigo", is an application programming interface in the .NET Framework for building connected, service-oriented applications.-The architectures:...
(WCF). The WCF is Microsoft's unified programming model for building service-oriented
Service-orientation
Service-orientation is a design paradigm to build computer software in the form of services. Like other design paradigms , service-orientation provides a governing approach to automate business logic as distributed systems...
applications, which enables developers to build secure, reliable, transacted solutions that integrate across platforms and interoperate with existing investments. To enable seamless integration of a provider into the KSP, and correspondingly seamless integration with third-party tools and applications, the providers must meet several conditions:
custom configuration settings stored in configuration files, rather than the source code;
standard .NET tracing and message logging to enable service monitoring and diagnostics;
standard Windows Management Interface performance counters for performance monitoring; and
document service contracts included using a service description language for better understanding and testing of the Keyword Service Provider.
These conditions help to simplify the task of developing well-integrated management applications, thus allowing business partners to produce best-of-breed, enterprise-scalable management solutions.
Stored procedures
Developers can write stored procedures (sprocs) using any .NET
.NET Framework
The .NET Framework is a software framework that runs primarily on Microsoft Windows. It includes a large library and supports several programming languages which allows language interoperability...
programming language. These procedures are executed on the Keyword Services Platform server, which hosts the Common Language Runtime
Common Language Runtime
The Common Language Runtime is the virtual machine component of Microsoft's .NET framework and is responsible for managing the execution of .NET programs. In a process known as just-in-time compilation, the CLR compiles the intermediate language code known as CIL into the machine instructions...
(CLR). Similar to a database sprocs, a KSP sproc is designed to enable developers to implement several types of business logic on the server side after retrieving result data from providers. KSP sprocs do not require configuration management or setup requirements.
Two types of stored procedures are supported: Managed Assembly Stored Procedure (MASP) and Common Language Runtime Stored Procedure (CLRSP). A MASP consists of a compiled .NET assembly containing a public interface exposed through the KSP as well as any dependent files. Once the MASP is uploaded to the KSP through its management interface, it becomes callable by KSP client programs. A CLRSP consists of a source file written in one of the supported CLR programming languages (C#, Visual Basic .NET
Visual Basic .NET
Visual Basic .NET , is an object-oriented computer programming language that can be viewed as an evolution of the classic Visual Basic , which is implemented on the .NET Framework...
Managed Extensions for C++ or just Managed C++ is a now deprecated Microsoft set of deviations from C++, including grammatical and syntactic extensions, keywords and attributes, to bring the C++ syntax and language to the .NET Framework...
, and others). The functionalities of the CLRSPs are exposed through a public interface defined in the source file. Once the CLRSP is deployed to KSP through its management interface, it is compiled on-demand by KSP and becomes callable by KSP client programs. Compared to database sprocs, KSP sprocs are object-oriented. A sproc may contain a set of related functions, or even identically named functions with different signatures.
Server Object Model and Shared Services
Keyword Services Platform Server Object Models and Shared Services enable KSP Service Providers and stored procedure developers to access server-side objects and functionalities easily and consistently. The object model consists of the following three collections:
Service providers: This collection enables callers to access server-side Service Provider objects by name, implementation interface, and/or class name. Once callers obtain the Service Provider object, all of the functionalities of the service provider are accessible through its public interface.
Stored procedures: This collection enables callers to access server-side Stored Procedure objects by name, implementation interface, and/or class name. Once callers obtain the Stored Procedure object, all of the functionalities of the stored procedure are accessible through its public interface.
Services: This collection enables callers to access server-side shared services by name, by implementation interface, and/or class name. Once callers obtain the shared service object, all of the functionalities of the shared service provider are accessible through its public interface.
Microsoft adCenter , is the division of the Microsoft Network responsible for MSN's advertising services. Microsoft adCenter provides pay per click advertisements. This is a service aimed at people who want to advertise a product...
A server farm or server cluster is a collection of computer servers usually maintained by an enterprise to accomplish server needs far beyond the capability of one machine. Server farms often have backup servers, which can take over the function of primary servers in the event of a primary server...
provides a scalable platform for keyword technologies. Each server in the farm can have different configuration to suit a variety of service providers and stored procedures. A dynamic service load balance server, a cloud server
Cloud computing
Cloud computing is the delivery of computing as a service rather than a product, whereby shared resources, software, and information are provided to computers and other devices as a utility over a network ....
, is the hub of the KSP server farm. When a KSP server is added to the server farm via the cloud server, all available keyword service providers and stored procedures are dynamically discovered and registered with the server. Any changes in the availability of the KSP server, as well as all its running service providers and stored procedures, are discovered and registered automatically with the server.
The cloud server distributes accesses to services running on a KSP server farm through its load balancer provider. The default implementation of the load balancer provider uses a round-robin scheduling
Round-robin scheduling
Round-robin is one of the simplest scheduling algorithms for processes in an operating system. As the term is generally used, time slices are assigned to each process in equal portions and in circular order, handling all processes without priority . Round-robin scheduling is simple, easy to...
approach. Over time, the server accumulates usage patterns and statistics of various service providers and stored procedures running on each KSP server in the farm. This information is used by the server to determine how to automatically deploy additional service providers and stored procedures. For example, if the Keyword Forecast provider is being used heavily in the server farm and the providers running on machine "A" are used lightly, the server will automatically deploy the Keyword Forecast provider to machine "A" and route requests to that machine to balance the load for the Keyword Forecast provider.
When a client application calls a service provider or stored procedure through the server, a KSP server with a matching service provider or stored procedure is selected by the load balancer provider, and the request is routed to the appropriate KSP server. If a server, service provider, or stored procedure in the KSP server farm is unavailable, it will be taken out of rotation by the load balancer automatically.
A data mart is the access layer of the data warehouse environment that is used to get data out to the users. The data mart is a subset of the data warehouse which is usually oriented to a specific business line or team.- Terminology :...
is a subset of an organizational data store, usually oriented to a specific purpose or major data subject, that may be distributed to support business needs. Many Keyword Services Platform providers require real-time database access. The database may contain a list of reference keywords, their corresponding traffic, most recent click-through data, and data mining model contents. This data is updated on a regular basis based on the provider's requirements.
Technology transfer
Keyword Services Platform's architecture permits agile development and rapid technology transfer by providing a platform for researchers to ship their research results to a live system quickly. The API defines the standard contract between the research models and developers. Researchers simply need to implement providers and deploy the providers into the selected set of KSP cloud server machines. The scope is limited, and thus very easy to use for live testing. Once the provider is live-tested and proven, KSP can switch to the default provider without any changes on the application side. This infrastructure enables researchers at Microsoft
Microsoft
Microsoft Corporation is an American public multinational corporation headquartered in Redmond, Washington, USA that develops, manufactures, licenses, and supports a wide range of products and services predominantly related to computing through its various product divisions...
and other academic settings to speed up innovation in keyword technology and deploy the latest research results to KSP consumers.
Microsoft Excel is a proprietary commercial spreadsheet application written and distributed by Microsoft for Microsoft Windows and Mac OS X. It features calculation, graphing tools, pivot tables, and a macro programming language called Visual Basic for Applications...
2007 that allows users to consume the Keyword Services Platform data directly via Excel rather than through the API. The add-in makes much of the keyword technology available directly through Excel. Essentially it is an example of the type of mashup
Mashup (digital)
A digital mashup usually is in reference to:1. Digital media content containing any or all of text, graphics, audio, video and animation drawn from pre-existing sources, to create a new derivative work...
and creative use of data that can be associated with the KSP. The add-in delivers features such as keyword extraction, suggestion, forecasting, monetization, etc.
Applications of the KSP
The Keyword Services Platform incorporates keyword technologies from Microsoft adCenter
Microsoft adCenter
Microsoft adCenter , is the division of the Microsoft Network responsible for MSN's advertising services. Microsoft adCenter provides pay per click advertisements. This is a service aimed at people who want to advertise a product...
Microsoft Research is the research division of Microsoft created in 1991 for developing various computer science ideas and integrating them into Microsoft products. It currently employs Turing Award winners C.A.R. Hoare, Butler Lampson, and Charles P...
groups. Keyword APIs can be consumed by third-party business applications from paid search, content advertisements, behavioral targeting
Behavioral targeting
Behavioral targeting is a technique used by online publishers and advertisers to increase the effectiveness of their campaigns.Behavioral targeting uses information collected on an individual's web-browsing behavior, such as the pages they have visited or the searches they have made, to select...
, presale business intelligence apps, and so on.
The KSP can be used in advertising campaign creation and management:
The Keyword Association provider can help advertisers generate a set of the most relevant keywords for a campaign, leading to more efficient planning and improved return on investment
ROI
-Roi:Roi may refer to:*Roi, a 1993 song by The Breeders in their album Last Splash*Roi Et and Roi Et province in Thailand*Roi Klein, an Israeli Defense major*Roi-Namur, an island in the Marshall Islands...
.
The Keyword Forecasting provider can help advertisers to understand traffic history and trends, and eventually help to manage an integrated campaign budget that makes seasonal allowances.
The Keyword Extraction provider can extract the important keywords on a publisher's webpage, helping to identify what advertisements should be served for that page, thus facilitating landing page
Landing page
In online marketing a landing page, sometimes known as a lead capture page, is a single web page that appears in response to clicking on an advertisement...
Display advertising is a type of advertising that typically contains text , logos, photographs or other images, location maps, and similar items. In periodicals, display advertising can appear on the same page as, or on the page adjacent to, general editorial content...
:
The Keyword Demographic and Geographic Distribution providers can help advertisers understand various customer segments and their keyword usage patterns, leading to more effectively targeted advertising and a decreased overall spend.
Keyword Association providers can help to expand existing customer segments to include other customers with similar interests based on language patterns.
ZhaoHui Tang, Jamie Maclennan, Pyungchul (Peter) Kim: Building data mining solutions with OLE DB for DM and XML for analysis. SIGMOD Record 34(2): 80-85 (2005)
ZhaoHui Tang, Jamie Maclennan: Data Mining with SQL Server 2008, Wiley, 2008.
