Open Directory Project
Encyclopedia
The Open Directory Project (ODP), also known as Dmoz (from directory.mozilla.org, its original domain name
), is a multilingual open content
directory
of World Wide Web
links. It is owned by Netscape
but it is constructed and maintained by a community
of volunteer editors.
ODP uses a hierarchical ontology scheme for organizing site listings. Listings on a similar topic are grouped into categories which can then include smaller categories.
and Bob Truel
in 1998 while they were both working as engineers for Sun Microsystems
. Chris Tolles, who worked at Sun Microsystems as the head of marketing for network security products, also signed on in 1998 as a co-founder of Gnuhoo along with co-founders Bryn Dole and Jeremy Wenokur. Skrenta had developed TASS
, an ancestor of tin
, the popular threaded Usenet
newsreader for Unix
systems. Coincidentally, the original category structure of the Gnuhoo directory was based loosely on the structure of Usenet newsgroups then in existence.
The Gnuhoo directory went live on June 5, 1998. After a Slashdot
article suggested that Gnuhoo had nothing in common with the spirit of free software
, for which the GNU
project was known, Richard Stallman and the Free Software Foundation
objected to the use of Gnu. So Gnuhoo was changed to NewHoo. Yahoo!
then objected to the use of "Hoo" in the name, prompting them to switch the name again. ZURL was the likely choice. However, before the switch to ZURL, NewHoo was acquired by Netscape Communications Corporation
in October 1998 and became the Open Directory Project. Netscape released the ODP data under the Open Directory License
. Netscape was acquired by AOL
shortly thereafter and ODP was one of the assets included in the acquisition. AOL later merged with Time-Warner.
By the time Netscape assumed stewardship, the Open Directory Project had about 100,000 URL
s indexed with contributions from about 4500 editors. On October 5, 1999, the number of URLs indexed by ODP reached one million. According to an unofficial estimate, the URLs in the Open Directory numbered 1.6 million in April 2000, surpassing those in the Yahoo! Directory
. ODP achieved the milestones of indexing two million URLs on August 14, 2000, three million listings on November 18, 2001 and four million on December 3, 2003.
From January 2006 the Open Directory published online reports to inform the public about the development of the project. The first report covered the year 2005. Monthly reports were issued subsequently until September 2006. These reports gave greater insight into the functioning of the directory than the simplified statistics given on the front page of the directory. The number of listings and categories cited on the front page include "Test" and "Bookmarks" categories but these are not included in the RDF dump offered to users. The total number of editors who have contributed to the directory as of March 31, 2007 was 75,151. There were about 7330 active editors during August 2006.
of the system that prevented editors from working on the directory until December 18, 2006. During that period, an older build of the directory was visible to the public. On January 13, 2007, the Site Suggestion and Update Listings forms were again made available. On January 26, 2007, weekly publication of RDF dumps resumed. To avoid future outages, the system now resides on a redundant configuration of two Intel-based servers.
edited by volunteers and sponsored by Go.com
and Zeal
emerged, both now defunct. These directories did not license their content for open content
distribution.
The concept of using a large-scale community of editors to compile online content has been successfully applied to other types of projects. ODP's editing model directly inspired three other open content volunteer projects: an open content restaurant directory known as ChefMoz
, an open content music directory known as MusicMoz, and an encyclopedia known as Open Site
.
Gnuhoo borrowed the basic outline for its initial ontology from Usenet
. In 1998, Rich Skrenta said, "I took a long list of groups and hand-edited them into a hierarchy." For example, the topic covered by the comp.ai.alife newsgroup was represented by the category Computers/AI/Artificial_Life. The original divisions were for Adult, Arts, Business, Computers, Games, Health, Home, News, Recreation, Reference, Regional, Science, Shopping, Society and Sports. While these fifteen top-level categories have remained intact, the ontology of second- and lower-level categories has undergone a gradual evolution; significant changes are initiated by discussion among editors and then implemented when consensus has been reached.
In July 1998, the directory became multilingual
with the addition of the World top-level category. The remainder of the directory lists only English language sites. By May 2005, seventy-five languages were represented. The growth rate of the non-English components of the directory has been greater than the English component since 2002. While the English component of the directory held almost 75% of the sites in 2003, the World level grew to over 1.5 million sites as of May 2005, forming roughly one-third of the directory. The ontology in non-English categories generally mirrors that of the English directory, although exceptions which reflect language differences are quite common.
Several of the top-level categories have unique characteristics. The Adult category is not present on the directory homepage but it is fully available in the RDF dump that ODP provides. While the bulk of the directory is categorized primarily by topic, the Regional category is categorized primarily by region. This has led many to view ODP as two parallel directories: Regional and Topical.
On November 14, 2000, a special directory within the Open Directory was created for people under 18 years of age. Key factors distinguishing this "Kids and Teens" area from the main directory are:
By May 2005, this portion of the Open Directory included over 32,000 site listings.
Since early 2004, the whole site has been in UTF-8
encoding. Prior to this, the encoding used to be ISO 8859-1 for English language categories and a language-dependent character set for other languages. The RDF dumps have been encoded in UTF-8 since early 2000.
Robozilla is a Web crawler
written to check the status of all sites listed in ODP. Periodically, Robozilla will flag sites which appear to have moved or disappeared and editors follow up to check the sites and take action. This process is critical for the directory in striving to achieve one of its founding goals: to reduce the link rot
in web directories. Shortly after each run, the sites marked with errors are automatically moved to the unreviewed queue where editors may investigate them when time permits.
Due to the popularity of the Open Directory and its resulting impact on search engine
rankings (See PageRank
), domains with lapsed registration that are listed on ODP have attracted domain hijacking
, an issue that has been addressed by regularly removing expired domains from the directory.
While corporate funding and staff for the ODP have diminished in recent years, volunteers have created editing tools such as linkcheckers to supplement Robozilla, category crawlers, spellcheckers, search tools that directly sift a recent RDF dump, bookmarklet
s to help automate some editing functions, mozilla based add-ons, and tools to help work through unreviewed queues.
, which requires a specific ODP attribution table on every Web page that uses the data.
The Open Directory License also includes a requirement that users of the data continually check the ODP site for updates and discontinue use and distribution of the data or works derived from the data once an update occurs. This restriction prompted the Free Software Foundation
to refer to the Open Directory License as a non-free documentation license, citing the right to redistribute a given version not being permanent and the requirement to check for changes to the license.
-like dump that is published on a dedicated download server, where an archive of previous versions is also available. New versions are usually generated weekly. An ODP editor has catalogued a number of bugs that are/were encountered when implementing the ODP RDF dump, including UTF-8
encoding errors (fixed since August 2004) and an RDF format that does not comply with the final RDF specification because ODP RDF generation was implemented before the RDF specification was finalized. So while today the so-called RDF dump is valid XML
, it is not strictly RDF but an ODP-specific format and as such, software to process the ODP RDF dump needs to take account of this.
s for many of the Web's largest search engines and portals, including Netscape Search, AOL Search, and Alexa
. Google Directory used ODP information, until being shuttered in July 2011.
Other uses are also made of ODP data. For example, in the spring of 2004 Overture
announced a search service for third parties combining Yahoo! Directory search results with ODP titles, descriptions and category metadata. The search engine Gigablast announced on May 12, 2005 its searchable copy of the Open Directory. The technology permits search of websites listed in specific categories, "in effect, instantly creating over 500,000 vertical search engines".
, the ODP listed 313 English-language Web sites that use ODP data as well as 238 sites in other languages. However, these figures do not reflect the full picture of use, as those sites that use ODP data without following the terms of the ODP license are not listed.
and otherwise give a sense of how the applicant would likely mesh with the ODP culture and mission. A majority of applications are rejected but reapplying is allowed and sometimes encouraged. The same standards apply to editors of all categories and subcategories.
ODP's editing model is a hierarchical one. Upon becoming editors, individuals will generally have editing permissions in only a small category. Once they have demonstrated basic editing skills in compliance with the Editing Guidelines, they are welcome to apply for additional editing privileges in either a broader category or in a category elsewhere in the directory. Mentorship relationships between editors are encouraged and internal forums provide a vehicle for new editors to ask questions.
ODP has its own internal forums, the contents of which are intended only for editors to communicate with each other primarily about editing topics. Access to the forums requires an editor account and editors are expected to keep the contents of these forums private.
Over time, senior editors may be granted additional privileges which reflect their editing experience and leadership within the editing community. The most straightforward are editall privileges which allow an editor to access all categories in the directory. Meta privileges additionally allow editors to perform tasks such as reviewing editor applications, setting category features and handling external and internal abuse reports. Cateditall privileges are similar to editall but only for a single directory category. Similarly, catmod privileges are similar to meta but only for a single directory category. Catmv privileges allow editors to make changes to directory ontology by moving or renaming categories. All of these privileges are granted by admins and staff, usually after discussion with meta editors.
In August 2004, a new level of privileges called admin was introduced. Administrator status was granted to a number of long serving metas by staff. Administrators have the ability to grant editall+ privileges to other editors and to approve new directory-wide policies, powers which had previously only been available to root (staff) editors. A full list of senior editors is available to the public, as is a listing of all current editors.
All ODP editors are expected to abide by ODP's Editing Guidelines. These guidelines describe editing basics: which types of sites may be listed and which may not; how site listings should be titled and described in a loosely consistent manner; conventions for the naming and building of categories; conflict of interest limitations on the editing of sites which the editor may own or otherwise be affiliated with; and a code of conduct within the community. Editors who are found to have violated these guidelines may be contacted by staff or senior editors, have their editing permissions cut back or lose their editing privileges entirely. ODP Guidelines are periodically revised after discussion in editor forums.
One result has been a gradual divergence between the ODP and other directories in the balance of content. The pay-for-inclusion model favours those able and willing to pay, so commercial sites tend to predominate in directories using it. Conversely, a directory manned by volunteers will reflect the aims and interests of those volunteers. The ODP lists a high proportion of informational and non-profit sites.
Another consequence of the free submission policy is that the ODP has enormous numbers of submissions still waiting for review. In large parts those consist of spam and incorrectly submitted sites. So the average processing time for a site submission has grown longer with each passing year. However the time taken cannot be predicted, since the variation is so great: a submission might be processed within hours or take several years. However, site suggestions are just one of many sources of new listings. Editors are under no obligation to check them for new listings and are actually encouraged to use other sources.
Early in the history of the ODP, its staff gave representatives of selected companies, such as Rolling Stone
or CNN
, editing access in order to list individual pages from their websites. Links to individual CNN articles have been added until 2004 and have been entirely removed from the directory in January 2008 due to being outdated and not considered worth the effort to maintain. Such experiments have not been repeated later.
At ODP's inception, there was little thought given to the idea of how ODP should be managed and there were no official forums
, guidelines or FAQ
s. In essence, ODP began as a free for all.
As time went on, the ODP Editor Forums became the de facto ODP parliament and when one of ODP's staff members would post an opinion in the forums, it would be considered an official ruling. Even so, ODP staff began to give trusted senior editors additional editing privileges, including the ability to approve new editor applications, which eventually led to a stratified hierarchy of duties and privileges among ODP editors, with ODP's paid staff having the final say regarding ODP's policies and procedures.
In the article Editor Removal Explained, ODP meta editor Arlarson states that "a great deal of confusion about the removal of editors from ODP results from false or misleading statements by former editors".
The ODP's confidentiality guidelines prohibit any current ODP editors in a position to know anything from discussing the reasons for specific editor removals. However, a generic list of reasons is for example given in the guidelines. In the past, this has led to removed ODP editors wondering why they cannot login at ODP to perform their editing work.
policies. According to their claims, staff used the excuse that their behaviour was uncivil to remove bothersome editors.
, folksonomies
began to appear. These people thought folksonomies, networks and directed graph
are more "natural" and easier to manage than hierarchies.
which is open source
, licensed under the Mozilla Public License
.
system. As of 2007, these forums are powered by a modified version of phpBB.
and the web server Apache
. Squid
web proxy server was also used but it was removed in August 2007 when the storage servers were reorganized. All these applications are open source.
). This has led to criticism from the aforementioned GNU project, many of whom also criticise the ODP content license.
As such, there have been some efforts to provide alternatives to ODP. These alternatives would allow communities of like-minded editors to set up and maintain their own open source/open content Web directories. However, no significant open source/open content alternative to ODP has emerged.
Domain name
A domain name is an identification string that defines a realm of administrative autonomy, authority, or control in the Internet. Domain names are formed by the rules and procedures of the Domain Name System ....
), is a multilingual open content
Open content
Open content or OpenContent is a neologism coined by David Wiley in 1998 which describes a creative work that others can copy or modify. The term evokes open source, which is a related concept in software....
directory
Web directory
A web directory or link directory is a directory on the World Wide Web. It specializes in linking to other web sites and categorizing those links....
of World Wide Web
World Wide Web
The World Wide Web is a system of interlinked hypertext documents accessed via the Internet...
links. It is owned by Netscape
Netscape
Netscape Communications is a US computer services company, best known for Netscape Navigator, its web browser. When it was an independent company, its headquarters were in Mountain View, California...
but it is constructed and maintained by a community
Virtual community
A virtual community is a social network of individuals who interact through specific media, potentially crossing geographical and political boundaries in order to pursue mutual interests or goals...
of volunteer editors.
ODP uses a hierarchical ontology scheme for organizing site listings. Listings on a similar topic are grouped into categories which can then include smaller categories.
Project information
ODP was founded in the United States as Gnuhoo by Rich SkrentaRich Skrenta
Richard "Rich" Skrenta is a computer programmer and Silicon Valley entrepreneur who created the search engine blekko....
and Bob Truel
Bob Truel
Bob Truel is a computer programmer. He met Rich Skrenta in ninth grade in Mt. Lebanon, Pennsylvania and has since co-founded several internet ventures with him and others, including the Open Directory Project with Bryn Dole, Chris Tolles, and Jeremy Wenokur in 1998, Newhoo in 1998, Topix.net with...
in 1998 while they were both working as engineers for Sun Microsystems
Sun Microsystems
Sun Microsystems, Inc. was a company that sold :computers, computer components, :computer software, and :information technology services. Sun was founded on February 24, 1982...
. Chris Tolles, who worked at Sun Microsystems as the head of marketing for network security products, also signed on in 1998 as a co-founder of Gnuhoo along with co-founders Bryn Dole and Jeremy Wenokur. Skrenta had developed TASS
TASS (software)
The Tass newsreader is an open source computer software package that provides a full screen threaded newsreader. It was written by Rich Skrenta, who did not like the rn newsreader and gained his inspirations from the Plato Notes system. The Tass newsreader was later modified by Iaian Lea, who added...
, an ancestor of tin
Tin (newsreader)
tin is an open source text-based and threaded news client, used to read and post messages on the USENET global communications network.-History:...
, the popular threaded Usenet
Usenet
Usenet is a worldwide distributed Internet discussion system. It developed from the general purpose UUCP architecture of the same name.Duke University graduate students Tom Truscott and Jim Ellis conceived the idea in 1979 and it was established in 1980...
newsreader for Unix
Unix
Unix is a multitasking, multi-user computer operating system originally developed in 1969 by a group of AT&T employees at Bell Labs, including Ken Thompson, Dennis Ritchie, Brian Kernighan, Douglas McIlroy, and Joe Ossanna...
systems. Coincidentally, the original category structure of the Gnuhoo directory was based loosely on the structure of Usenet newsgroups then in existence.
The Gnuhoo directory went live on June 5, 1998. After a Slashdot
Slashdot
Slashdot is a technology-related news website owned by Geeknet, Inc. The site, which bills itself as "News for Nerds. Stuff that Matters", features user-submitted and ‑evaluated current affairs news stories about science- and technology-related topics. Each story has a comments section...
article suggested that Gnuhoo had nothing in common with the spirit of free software
Free software
Free software, software libre or libre software is software that can be used, studied, and modified without restriction, and which can be copied and redistributed in modified or unmodified form either without restriction, or with restrictions that only ensure that further recipients can also do...
, for which the GNU
GNU
GNU is a Unix-like computer operating system developed by the GNU project, ultimately aiming to be a "complete Unix-compatible software system"...
project was known, Richard Stallman and the Free Software Foundation
Free Software Foundation
The Free Software Foundation is a non-profit corporation founded by Richard Stallman on 4 October 1985 to support the free software movement, a copyleft-based movement which aims to promote the universal freedom to create, distribute and modify computer software...
objected to the use of Gnu. So Gnuhoo was changed to NewHoo. Yahoo!
Yahoo!
Yahoo! Inc. is an American multinational internet corporation headquartered in Sunnyvale, California, United States. The company is perhaps best known for its web portal, search engine , Yahoo! Directory, Yahoo! Mail, Yahoo! News, Yahoo! Groups, Yahoo! Answers, advertising, online mapping ,...
then objected to the use of "Hoo" in the name, prompting them to switch the name again. ZURL was the likely choice. However, before the switch to ZURL, NewHoo was acquired by Netscape Communications Corporation
Netscape
Netscape Communications is a US computer services company, best known for Netscape Navigator, its web browser. When it was an independent company, its headquarters were in Mountain View, California...
in October 1998 and became the Open Directory Project. Netscape released the ODP data under the Open Directory License
Open Directory License
The Open Directory License is a license that was used by the Open Directory Project for its content. It is like many open source licenses, which are used for many types of software and sometimes its corresponding documentation. Examples of this strategy may include or ChefMoz, MusicMoz and...
. Netscape was acquired by AOL
AOL
AOL Inc. is an American global Internet services and media company. AOL is headquartered at 770 Broadway in New York. Founded in 1983 as Control Video Corporation, it has franchised its services to companies in several nations around the world or set up international versions of its services...
shortly thereafter and ODP was one of the assets included in the acquisition. AOL later merged with Time-Warner.
By the time Netscape assumed stewardship, the Open Directory Project had about 100,000 URL
Uniform Resource Locator
In computing, a uniform resource locator or universal resource locator is a specific character string that constitutes a reference to an Internet resource....
s indexed with contributions from about 4500 editors. On October 5, 1999, the number of URLs indexed by ODP reached one million. According to an unofficial estimate, the URLs in the Open Directory numbered 1.6 million in April 2000, surpassing those in the Yahoo! Directory
Yahoo! Directory
The Yahoo! Directory is a web directory that rivals the Open Directory Project in size. The directory was Yahoo!'s first offering. When Yahoo! changed to crawler-based listings for its main results in October 2002, the human-edited directory's significance dropped, but it was still being updated in...
. ODP achieved the milestones of indexing two million URLs on August 14, 2000, three million listings on November 18, 2001 and four million on December 3, 2003.
From January 2006 the Open Directory published online reports to inform the public about the development of the project. The first report covered the year 2005. Monthly reports were issued subsequently until September 2006. These reports gave greater insight into the functioning of the directory than the simplified statistics given on the front page of the directory. The number of listings and categories cited on the front page include "Test" and "Bookmarks" categories but these are not included in the RDF dump offered to users. The total number of editors who have contributed to the directory as of March 31, 2007 was 75,151. There were about 7330 active editors during August 2006.
System failure and editing outage, October to December 2006
On October 20, 2006, the ODP's main server suffered a catastrophic failureCatastrophic failure
A catastrophic failure is a sudden and total failure of some system from which recovery is impossible. Catastrophic failures often lead to cascading systems failure....
of the system that prevented editors from working on the directory until December 18, 2006. During that period, an older build of the directory was visible to the public. On January 13, 2007, the Site Suggestion and Update Listings forms were again made available. On January 26, 2007, weekly publication of RDF dumps resumed. To avoid future outages, the system now resides on a redundant configuration of two Intel-based servers.
Competing and spinoff projects
As the ODP became more widely known, two other major web directoriesWeb directory
A web directory or link directory is a directory on the World Wide Web. It specializes in linking to other web sites and categorizing those links....
edited by volunteers and sponsored by Go.com
Go.com
Go.com is a web portal first launched by Jeff Gold, and now operated by the Walt Disney Internet Group, which is a part of The Walt Disney Company. The portal includes content from ABC News, ESPN, and FamilyFun.com, all of which are associated with Disney and are hosted under a .go.com name...
and Zeal
Zeal (web)
Zeal was a volunteer-built web directory, first appearing in 1999, and then acquired by LookSmart in October 2000 for $20 million. Zeal combined the work of Looksmart's paid editors with that of volunteers who profiled websites and placed them in a hierarchy of subcategories...
emerged, both now defunct. These directories did not license their content for open content
Open content
Open content or OpenContent is a neologism coined by David Wiley in 1998 which describes a creative work that others can copy or modify. The term evokes open source, which is a related concept in software....
distribution.
The concept of using a large-scale community of editors to compile online content has been successfully applied to other types of projects. ODP's editing model directly inspired three other open content volunteer projects: an open content restaurant directory known as ChefMoz
ChefMoz
Chef Moz, an offshoot of the Open Directory Project , was an open content directory of World Wide Web links of restaurants. The website was constructed and maintained by a community of volunteer editors, and owned by Netscape....
, an open content music directory known as MusicMoz, and an encyclopedia known as Open Site
Open Site
Open-Site, the Open Encyclopedia Project, is a free internet encyclopedia founded in 2002 by Michael J. Flickinger in an effort to build a free categorized community-built encyclopedia, inspired by the Open Directory Project...
.
Content
Usenet
Usenet is a worldwide distributed Internet discussion system. It developed from the general purpose UUCP architecture of the same name.Duke University graduate students Tom Truscott and Jim Ellis conceived the idea in 1979 and it was established in 1980...
. In 1998, Rich Skrenta said, "I took a long list of groups and hand-edited them into a hierarchy." For example, the topic covered by the comp.ai.alife newsgroup was represented by the category Computers/AI/Artificial_Life. The original divisions were for Adult, Arts, Business, Computers, Games, Health, Home, News, Recreation, Reference, Regional, Science, Shopping, Society and Sports. While these fifteen top-level categories have remained intact, the ontology of second- and lower-level categories has undergone a gradual evolution; significant changes are initiated by discussion among editors and then implemented when consensus has been reached.
In July 1998, the directory became multilingual
Multilingualism
Multilingualism is the act of using, or promoting the use of, multiple languages, either by an individual speaker or by a community of speakers. Multilingual speakers outnumber monolingual speakers in the world's population. Multilingualism is becoming a social phenomenon governed by the needs of...
with the addition of the World top-level category. The remainder of the directory lists only English language sites. By May 2005, seventy-five languages were represented. The growth rate of the non-English components of the directory has been greater than the English component since 2002. While the English component of the directory held almost 75% of the sites in 2003, the World level grew to over 1.5 million sites as of May 2005, forming roughly one-third of the directory. The ontology in non-English categories generally mirrors that of the English directory, although exceptions which reflect language differences are quite common.
Several of the top-level categories have unique characteristics. The Adult category is not present on the directory homepage but it is fully available in the RDF dump that ODP provides. While the bulk of the directory is categorized primarily by topic, the Regional category is categorized primarily by region. This has led many to view ODP as two parallel directories: Regional and Topical.
On November 14, 2000, a special directory within the Open Directory was created for people under 18 years of age. Key factors distinguishing this "Kids and Teens" area from the main directory are:
- stricter guidelines which limit the listing of sites to those which are targeted or "appropriate" for people under 18 years of age;
- category names as well as site descriptions use vocabulary which is "age appropriateAge appropriateAge appropriate refers to a developmental concept whereby certain activities may be deemed appropriate or inappropriate to a child's "stage" or level of development....
"; - age tags on each listing distinguish content appropriate for kids (age 12 and under), teens (13 to 15 years old) and mature teens (16 to 18 years old);
- Kids and Teens content is available as a separate RDF dump;
- editing permissions are such that the community is parallel to that of the Open Directory.
By May 2005, this portion of the Open Directory included over 32,000 site listings.
Since early 2004, the whole site has been in UTF-8
UTF-8
UTF-8 is a multibyte character encoding for Unicode. Like UTF-16 and UTF-32, UTF-8 can represent every character in the Unicode character set. Unlike them, it is backward-compatible with ASCII and avoids the complications of endianness and byte order marks...
encoding. Prior to this, the encoding used to be ISO 8859-1 for English language categories and a language-dependent character set for other languages. The RDF dumps have been encoded in UTF-8 since early 2000.
Maintenance
Directory listings are maintained by editors. While some editors focus on the addition of new listings, others focus on maintaining the existing listings. This includes tasks such as the editing of individual listings to correct spelling and/or grammatical errors, as well as monitoring the status of linked sites. Still others go through site submissions to remove spam and duplicate submissions.Robozilla is a Web crawler
Web crawler
A Web crawler is a computer program that browses the World Wide Web in a methodical, automated manner or in an orderly fashion. Other terms for Web crawlers are ants, automatic indexers, bots, Web spiders, Web robots, or—especially in the FOAF community—Web scutters.This process is called Web...
written to check the status of all sites listed in ODP. Periodically, Robozilla will flag sites which appear to have moved or disappeared and editors follow up to check the sites and take action. This process is critical for the directory in striving to achieve one of its founding goals: to reduce the link rot
Link rot
Link rot , also known as link death or link breaking is an informal term for the process by which, either on individual websites or the Internet in general, increasing numbers of links point to web pages, servers or other resources that have become permanently unavailable...
in web directories. Shortly after each run, the sites marked with errors are automatically moved to the unreviewed queue where editors may investigate them when time permits.
Due to the popularity of the Open Directory and its resulting impact on search engine
Web search engine
A web search engine is designed to search for information on the World Wide Web and FTP servers. The search results are generally presented in a list of results often referred to as SERPS, or "search engine results pages". The information may consist of web pages, images, information and other...
rankings (See PageRank
PageRank
PageRank is a link analysis algorithm, named after Larry Page and used by the Google Internet search engine, that assigns a numerical weighting to each element of a hyperlinked set of documents, such as the World Wide Web, with the purpose of "measuring" its relative importance within the set...
), domains with lapsed registration that are listed on ODP have attracted domain hijacking
Domain hijacking
Domain hijacking or domain theft is the act of changing the registration of a domain name without the permission of its original registrant....
, an issue that has been addressed by regularly removing expired domains from the directory.
While corporate funding and staff for the ODP have diminished in recent years, volunteers have created editing tools such as linkcheckers to supplement Robozilla, category crawlers, spellcheckers, search tools that directly sift a recent RDF dump, bookmarklet
Bookmarklet
A bookmarklet is Unobtrusive JavaScript stored as the URL of a bookmark in a web browser or as a hyperlink on a web page. The term is a portmanteau of the terms bookmark and applet, however, an applet is not to be confused with a bookmarklet just as JavaScript is not to be confused with Java...
s to help automate some editing functions, mozilla based add-ons, and tools to help work through unreviewed queues.
License and requirements
ODP data is made available for open content distribution under the terms of the Open Directory LicenseOpen Directory License
The Open Directory License is a license that was used by the Open Directory Project for its content. It is like many open source licenses, which are used for many types of software and sometimes its corresponding documentation. Examples of this strategy may include or ChefMoz, MusicMoz and...
, which requires a specific ODP attribution table on every Web page that uses the data.
The Open Directory License also includes a requirement that users of the data continually check the ODP site for updates and discontinue use and distribution of the data or works derived from the data once an update occurs. This restriction prompted the Free Software Foundation
Free Software Foundation
The Free Software Foundation is a non-profit corporation founded by Richard Stallman on 4 October 1985 to support the free software movement, a copyleft-based movement which aims to promote the universal freedom to create, distribute and modify computer software...
to refer to the Open Directory License as a non-free documentation license, citing the right to redistribute a given version not being permanent and the requirement to check for changes to the license.
RDF dumps
ODP data is made available through an RDFResource Description Framework
The Resource Description Framework is a family of World Wide Web Consortium specifications originally designed as a metadata data model...
-like dump that is published on a dedicated download server, where an archive of previous versions is also available. New versions are usually generated weekly. An ODP editor has catalogued a number of bugs that are/were encountered when implementing the ODP RDF dump, including UTF-8
UTF-8
UTF-8 is a multibyte character encoding for Unicode. Like UTF-16 and UTF-32, UTF-8 can represent every character in the Unicode character set. Unlike them, it is backward-compatible with ASCII and avoids the complications of endianness and byte order marks...
encoding errors (fixed since August 2004) and an RDF format that does not comply with the final RDF specification because ODP RDF generation was implemented before the RDF specification was finalized. So while today the so-called RDF dump is valid XML
XML
Extensible Markup Language is a set of rules for encoding documents in machine-readable form. It is defined in the XML 1.0 Specification produced by the W3C, and several other related specifications, all gratis open standards....
, it is not strictly RDF but an ODP-specific format and as such, software to process the ODP RDF dump needs to take account of this.
Content users
ODP data powers the core directory serviceDirectory service
A directory service is the software system that stores, organizes and provides access to information in a directory. In software engineering, a directory is a map between names and values. It allows the lookup of values given a name, similar to a dictionary...
s for many of the Web's largest search engines and portals, including Netscape Search, AOL Search, and Alexa
Alexa Internet
Alexa Internet, Inc. is a California-based subsidiary company of Amazon.com that is known for its toolbar and Web site. Once installed, the toolbar collects data on browsing behavior which is transmitted to the Web site where it is stored and analyzed and is the basis for the company's Web traffic...
. Google Directory used ODP information, until being shuttered in July 2011.
Other uses are also made of ODP data. For example, in the spring of 2004 Overture
Yahoo! Search Marketing
Yahoo! Search Marketing is a keyword-based "Pay per click" or "Sponsored search" Internet advertising service provided by Yahoo!.Yahoo began offering this service after acquiring Overture Services, Inc....
announced a search service for third parties combining Yahoo! Directory search results with ODP titles, descriptions and category metadata. The search engine Gigablast announced on May 12, 2005 its searchable copy of the Open Directory. The technology permits search of websites listed in specific categories, "in effect, instantly creating over 500,000 vertical search engines".
, the ODP listed 313 English-language Web sites that use ODP data as well as 238 sites in other languages. However, these figures do not reflect the full picture of use, as those sites that use ODP data without following the terms of the ODP license are not listed.
Policies and procedures
There are restrictions imposed on who can become an ODP editor. The primary gatekeeping mechanism is an editor application process wherein editor candidates demonstrate their editing abilities, disclose affiliations that might pose a conflict of interestConflict of interest
A conflict of interest occurs when an individual or organization is involved in multiple interests, one of which could possibly corrupt the motivation for an act in the other....
and otherwise give a sense of how the applicant would likely mesh with the ODP culture and mission. A majority of applications are rejected but reapplying is allowed and sometimes encouraged. The same standards apply to editors of all categories and subcategories.
ODP's editing model is a hierarchical one. Upon becoming editors, individuals will generally have editing permissions in only a small category. Once they have demonstrated basic editing skills in compliance with the Editing Guidelines, they are welcome to apply for additional editing privileges in either a broader category or in a category elsewhere in the directory. Mentorship relationships between editors are encouraged and internal forums provide a vehicle for new editors to ask questions.
ODP has its own internal forums, the contents of which are intended only for editors to communicate with each other primarily about editing topics. Access to the forums requires an editor account and editors are expected to keep the contents of these forums private.
Over time, senior editors may be granted additional privileges which reflect their editing experience and leadership within the editing community. The most straightforward are editall privileges which allow an editor to access all categories in the directory. Meta privileges additionally allow editors to perform tasks such as reviewing editor applications, setting category features and handling external and internal abuse reports. Cateditall privileges are similar to editall but only for a single directory category. Similarly, catmod privileges are similar to meta but only for a single directory category. Catmv privileges allow editors to make changes to directory ontology by moving or renaming categories. All of these privileges are granted by admins and staff, usually after discussion with meta editors.
In August 2004, a new level of privileges called admin was introduced. Administrator status was granted to a number of long serving metas by staff. Administrators have the ability to grant editall+ privileges to other editors and to approve new directory-wide policies, powers which had previously only been available to root (staff) editors. A full list of senior editors is available to the public, as is a listing of all current editors.
All ODP editors are expected to abide by ODP's Editing Guidelines. These guidelines describe editing basics: which types of sites may be listed and which may not; how site listings should be titled and described in a loosely consistent manner; conventions for the naming and building of categories; conflict of interest limitations on the editing of sites which the editor may own or otherwise be affiliated with; and a code of conduct within the community. Editors who are found to have violated these guidelines may be contacted by staff or senior editors, have their editing permissions cut back or lose their editing privileges entirely. ODP Guidelines are periodically revised after discussion in editor forums.
Site submissions
One of the original motivations for forming Gnuhoo/Newhoo/ODP was the frustration that many people experienced in getting their sites listed on Yahoo! Directory. However Yahoo! has since implemented a paid service for timely consideration of site submissions. That lead has been followed by many other directories. Some accept no free submissions at all. By contrast the ODP has maintained its policy of free site submissions for all types of site—the only one of the major general directories to do so.One result has been a gradual divergence between the ODP and other directories in the balance of content. The pay-for-inclusion model favours those able and willing to pay, so commercial sites tend to predominate in directories using it. Conversely, a directory manned by volunteers will reflect the aims and interests of those volunteers. The ODP lists a high proportion of informational and non-profit sites.
Another consequence of the free submission policy is that the ODP has enormous numbers of submissions still waiting for review. In large parts those consist of spam and incorrectly submitted sites. So the average processing time for a site submission has grown longer with each passing year. However the time taken cannot be predicted, since the variation is so great: a submission might be processed within hours or take several years. However, site suggestions are just one of many sources of new listings. Editors are under no obligation to check them for new listings and are actually encouraged to use other sources.
Controversy and criticism
There have long been allegations that volunteer ODP editors give favorable treatment to their own websites while concomitantly thwarting the good faith efforts of their competition. Such allegations are fielded by ODP's staff and meta editors, who have the authority to take disciplinary action against volunteer editors who are suspected of engaging in abusive editing practices. In 2003, ODP introduced a new Public Abuse Report System that allows members of the general public to report and track allegations of abusive editor conduct using an online form. Uninhibited discussion of ODP's purported shortcomings has become more common on mainstream Webmaster discussion forums. Although site policies suggest that an individual site should be submitted to only one category, as of October 2007, Topix.com, a news aggregation site operated by ODP founder Rich Skrenta, has more than 10,000 listings.Early in the history of the ODP, its staff gave representatives of selected companies, such as Rolling Stone
Rolling Stone
Rolling Stone is a US-based magazine devoted to music, liberal politics, and popular culture that is published every two weeks. Rolling Stone was founded in San Francisco in 1967 by Jann Wenner and music critic Ralph J...
or CNN
CNN
Cable News Network is a U.S. cable news channel founded in 1980 by Ted Turner. Upon its launch, CNN was the first channel to provide 24-hour television news coverage, and the first all-news television channel in the United States...
, editing access in order to list individual pages from their websites. Links to individual CNN articles have been added until 2004 and have been entirely removed from the directory in January 2008 due to being outdated and not considered worth the effort to maintain. Such experiments have not been repeated later.
Ownership and management
Underlying some controversy surrounding ODP is its ownership and management. Some of the original GnuHoo volunteers felt that they had been deceived into joining a commercial enterprise. To varying degrees, those complaints have continued up until the present.At ODP's inception, there was little thought given to the idea of how ODP should be managed and there were no official forums
Internet forum
An Internet forum, or message board, is an online discussion site where people can hold conversations in the form of posted messages. They differ from chat rooms in that messages are at least temporarily archived...
, guidelines or FAQ
FAQ
Frequently asked questions are listed questions and answers, all supposed to be commonly asked in some context, and pertaining to a particular topic. "FAQ" is usually pronounced as an initialism rather than an acronym, but an acronym form does exist. Since the acronym FAQ originated in textual...
s. In essence, ODP began as a free for all.
As time went on, the ODP Editor Forums became the de facto ODP parliament and when one of ODP's staff members would post an opinion in the forums, it would be considered an official ruling. Even so, ODP staff began to give trusted senior editors additional editing privileges, including the ability to approve new editor applications, which eventually led to a stratified hierarchy of duties and privileges among ODP editors, with ODP's paid staff having the final say regarding ODP's policies and procedures.
Editor removal procedures
ODP's editor removal procedures are overseen by ODP's staff and meta editors. According to ODP's official editorial guidelines, editors are removed for abusive editing practices or uncivil behaviour. Discussions that may result in disciplinary action against volunteer editors take place in a private forum which can only be accessed by ODP's staff, meta editors and volunteer editors who are being discussed are not given notice that such proceedings are taking place. Some people find this arrangement distasteful, wanting instead a discussion modelled more like a trial held in the U.S. judicial system.In the article Editor Removal Explained, ODP meta editor Arlarson states that "a great deal of confusion about the removal of editors from ODP results from false or misleading statements by former editors".
The ODP's confidentiality guidelines prohibit any current ODP editors in a position to know anything from discussing the reasons for specific editor removals. However, a generic list of reasons is for example given in the guidelines. In the past, this has led to removed ODP editors wondering why they cannot login at ODP to perform their editing work.
Allegations that editors are removed for criticizing policies
David F. Prenatt, Jr. (former ODP editor netesq) and the former editor known by the alias The Cunctator claim to have been removed for disagreeing with staff about changes to the policies, with special regard to the ODP's copyrightCopyright
Copyright is a legal concept, enacted by most governments, giving the creator of an original work exclusive rights to it, usually for a limited time...
policies. According to their claims, staff used the excuse that their behaviour was uncivil to remove bothersome editors.
Blacklisting allegations
Senior ODP editors have the ability to attach "warning" or "do not list" notes to individual domains but no editor has the unilateral ability to block certain sites from being listed. Sites with these notes might still be listed and at times notes are removed after some discussion.Hierarchical structure
Recently criticism of ODP's hierarchical structure emerged. Many believe hierarchical directories are too complicated. As the recent emergence of Web 2.0Web 2.0
The term Web 2.0 is associated with web applications that facilitate participatory information sharing, interoperability, user-centered design, and collaboration on the World Wide Web...
, folksonomies
Folksonomy
A folksonomy is a system of classification derived from the practice and method of collaboratively creating and managing tags to annotate and categorize content; this practice is also known as collaborative tagging, social classification, social indexing, and social tagging...
began to appear. These people thought folksonomies, networks and directed graph
Directed graph
A directed graph or digraph is a pair G= of:* a set V, whose elements are called vertices or nodes,...
are more "natural" and easier to manage than hierarchies.
Search
The ODPSearch software is a derivative version of IsearchIsearch
Isearch is open-source text retrieval software first developed in 1994 as part of the Isite Z39.50 information framework. The project started at the Clearinghouse for Networked Information Discovery and Retrieval of the North Carolina supercomputing center MCNC and funded by the National Science...
which is open source
Open source
The term open source describes practices in production and development that promote access to the end product's source materials. Some consider open source a philosophy, others consider it a pragmatic methodology...
, licensed under the Mozilla Public License
Mozilla Public License
The Mozilla Public License is a free and open source software license. Version 1.0 was developed by Mitchell Baker when she worked as a lawyer at Netscape Communications Corporation and version 1.1 at the Mozilla Foundation...
.
Editor forums
The ODP Editor Forums were originally run on software that was based on the proprietary Ultimate Bulletin Board system. In June 2003, they switched to the open source phpBBPhpBB
phpBB is a popular Internet forum package written in the PHP scripting language. The name "phpBB" is an abbreviation of PHP Bulletin Board...
system. As of 2007, these forums are powered by a modified version of phpBB.
Bug tracking
The bug tracking software used by the ODP is BugzillaBugzilla
Bugzilla is a Web-based general-purpose bugtracker and testing tool originally developed and used by the Mozilla project, and licensed under the Mozilla Public License....
and the web server Apache
Apache HTTP Server
The Apache HTTP Server, commonly referred to as Apache , is web server software notable for playing a key role in the initial growth of the World Wide Web. In 2009 it became the first web server software to surpass the 100 million website milestone...
. Squid
Squid cache
Squid is a proxy server and web cache daemon. It has a wide variety of uses, from speeding up a web server by caching repeated requests; to caching web, DNS and other computer network lookups for a group of people sharing network resources; to aiding security by filtering traffic...
web proxy server was also used but it was removed in August 2007 when the storage servers were reorganized. All these applications are open source.
Interface
The ODP database/editing software is closed source (although Richard Skrenta of ODP did say in June 1998 that he was considering licensing it under the GNU General Public LicenseGNU General Public License
The GNU General Public License is the most widely used free software license, originally written by Richard Stallman for the GNU Project....
). This has led to criticism from the aforementioned GNU project, many of whom also criticise the ODP content license.
As such, there have been some efforts to provide alternatives to ODP. These alternatives would allow communities of like-minded editors to set up and maintain their own open source/open content Web directories. However, no significant open source/open content alternative to ODP has emerged.